Plants Seedlings Classification

Context

In recent times, the field of agriculture has been in urgent need of modernizing, since the amount of manual work people need to put in to check if plants are growing correctly is still highly extensive. Despite several advances in agricultural technology, people working in the agricultural industry still need to have the ability to sort and recognize different plants and weeds, which takes a lot of time and effort in the long term.

The potential is ripe for this trillion-dollar industry to be greatly impacted by technological innovations that cut down on the requirement for manual labor, and this is where Artificial Intelligence can benefit the workers in this field, as the time and energy required to identify plant seedlings will be greatly shortened by the use of AI and Deep Learning. The ability to do so far more efficiently and even more effectively than experienced manual labor could lead to better crop yields, the freeing up of human involvement for higher-order agricultural decision making, and in the long term will result in more sustainable environmental practices in agriculture as well.

AI Generated Image

Objective

To classify the plant seedlings into their respective categories.

Dataset

The Aarhus University Signal Processing group, in collaboration with the University of Southern Denmark, has provided the data containing images of unique plants belonging to 12 different species.

There are 12 plant species (categories) in the dataset:

• Black-grass
• Charlock
• Cleavers
• Common Chickweed
• Common Wheat
• Fat Hen
• Loose Silky-bent
• Maize
• Scentless Mayweed
• Shepherds Purse
• Small-flowered Cranesbill
• Sugar beet

Setup

Setup Google Drive

#connect to google drive
from google.colab import drive
drive.mount('/content/drive')

Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).

#parent directory where images are stored
image_file = '/content/drive/MyDrive/Projects/ML/Data/plantSeedlings_images.npy' #npy file containing images
label_file = '/content/drive/MyDrive/Projects/ML/Data/plantSeedlings_Labels.csv' #csv file containing the labels for each image of the npy file

Import Libraries and Basics

#visualkeras is a library to visualize cnn architectures
!pip install visualkeras

Requirement already satisfied: visualkeras in /usr/local/lib/python3.10/dist-packages (0.0.2)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from visualkeras) (9.4.0)
Requirement already satisfied: numpy>=1.18.1 in /usr/local/lib/python3.10/dist-packages (from visualkeras) (1.23.5)
Requirement already satisfied: aggdraw>=1.3.11 in /usr/local/lib/python3.10/dist-packages (from visualkeras) (1.3.16)

#library for creating data paths
import os

#library for randomly selecting data points
import random

#library for performing numerical computations
import numpy as np

#library for data manipulation and analysis
import pandas as pd

#library for data visualization
import seaborn as sns

#library for creating and showing data
import matplotlib.pyplot as plt

#library for reading and showing images
import matplotlib.image as mpimg

#Python Imaging Library to handle image files
from PIL import Image

#OpenCv for image and video processing
import cv2

#sklearn
from sklearn.model_selection import train_test_split
from sklearn.utils import class_weight
from sklearn.utils.class_weight import compute_sample_weight
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
from sklearn.preprocessing import LabelBinarizer

#tensorflow
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.preprocessing.image import img_to_array, load_img

#importing all the required sub-modules from Keras
from keras import backend
from keras.utils import to_categorical
from keras.models import Sequential, Model
from keras.applications.vgg16 import VGG16
from keras.preprocessing.image import ImageDataGenerator

from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, BatchNormalization, Dropout
from keras import losses, optimizers
from keras.optimizers import RMSprop,Adam,SGD #optimiers for optimizing the model
from keras.callbacks import EarlyStopping, ModelCheckpoint  #regularization method to prevent the overfitting
from keras.utils.vis_utils import plot_model

#library to visualize model architecture
import visualkeras

#ignore warnings that may occur due to deprecations
import warnings
warnings.filterwarnings("ignore")

Data Analysis and Data Treatment

First we will load the image_files and their corresponding labels.

#load the numpy array
img_array = np.load(image_file)

#visualize contents of array, index 0
img_array[0]

array([[[ 35,  52,  78],
        [ 36,  49,  76],
        [ 31,  45,  69],
        ...,
        [ 78,  95, 114],
        [ 76,  93, 110],
        [ 80,  95, 109]],

       [[ 33,  46,  68],
        [ 37,  50,  73],
        [ 48,  65,  83],
        ...,
        [ 81,  96, 113],
        [ 74,  89, 105],
        [ 83,  95, 109]],

       [[ 34,  50,  68],
        [ 35,  52,  72],
        [ 70,  85, 101],
        ...,
        [ 83,  97, 112],
        [ 79,  94, 108],
        [ 79,  94, 107]],

       ...,

       [[ 35,  50,  69],
        [ 42,  57,  73],
        [ 42,  57,  72],
        ...,
        [ 60,  76,  92],
        [ 67,  81,  97],
        [ 64,  77,  95]],

       [[ 36,  52,  67],
        [ 48,  63,  78],
        [ 41,  57,  73],
        ...,
        [ 44,  66,  83],
        [ 58,  76,  91],
        [ 57,  74,  90]],

       [[ 44,  58,  70],
        [ 43,  57,  73],
        [ 40,  55,  72],
        ...,
        [ 41,  70,  92],
        [ 55,  78,  97],
        [ 61,  79,  96]]], dtype=uint8)

Shape of Array images

#shape of the image array
img_array.shape

(4750, 128, 128, 3)

The image array contains 4,750 records. Each record has a length of 128 rows and 128 columns (resolution in pixels). And each element has three values representing Red, Green, Blue (RGB).

#read the labels and store in a dataframe
label_df = pd.read_csv(label_file)

#display the head of the labels dataset
label_df.head(10)

	Label
0	Small-flowered Cranesbill
1	Small-flowered Cranesbill
2	Small-flowered Cranesbill
3	Small-flowered Cranesbill
4	Small-flowered Cranesbill
5	Small-flowered Cranesbill
6	Small-flowered Cranesbill
7	Small-flowered Cranesbill
8	Small-flowered Cranesbill
9	Small-flowered Cranesbill

Shape of dataframe Labels

#print shape of label dataframe
label_df.shape

(4750, 1)

There are 4750 records in total.

#display information of the label dataframe and its data type
label_df['Label'].info

<bound method Series.info of 0       Small-flowered Cranesbill
1       Small-flowered Cranesbill
2       Small-flowered Cranesbill
3       Small-flowered Cranesbill
4       Small-flowered Cranesbill
                  ...            
4745             Loose Silky-bent
4746             Loose Silky-bent
4747             Loose Silky-bent
4748             Loose Silky-bent
4749             Loose Silky-bent
Name: Label, Length: 4750, dtype: object>

We should convert 'Label' to a string type because later we will convert it to one-hot vectors. To convert to one-hot vectors, we will use 'to_categorical' which expects an array of integers or numeric values. Otherwise type may throw an error.

#convert to string type
label_df['Label'] = label_df['Label'].astype(str)

Number of Unique Classes

#Number of unique classes (labels)
num_classes = label_df.nunique()[0]
print("The number of seed classes is ",num_classes)

The number of seed classes is  12

Name of Classes

#create an array containing the labels of classes
name_classes = label_df['Label'].unique()
name_classes

array(['Small-flowered Cranesbill', 'Fat Hen', 'Shepherds Purse',
       'Common wheat', 'Common Chickweed', 'Charlock', 'Cleavers',
       'Scentless Mayweed', 'Sugar beet', 'Maize', 'Black-grass',
       'Loose Silky-bent'], dtype=object)

Distribution of Classes

#visualize the seedlings label
figsize=(14, 8)
count_plot = sns.countplot(data=label_df, x="Label")

#rotate x-axis labels
count_plot.set_xticklabels(count_plot.get_xticklabels(), rotation=90)
count_plot.set(xlabel='seed')

plt.show()

#Number of items per categories
label_df.value_counts()

Label                    
Loose Silky-bent             654
Common Chickweed             611
Scentless Mayweed            516
Small-flowered Cranesbill    496
Fat Hen                      475
Charlock                     390
Sugar beet                   385
Cleavers                     287
Black-grass                  263
Shepherds Purse              231
Common wheat                 221
Maize                        221
dtype: int64

The data is imbalanced. Class Maize and Common Wheat are roughly 1/3 the size of Loose Silky-bent.

Plotting images

#convert dataframe to array
label_array = label_df.to_numpy().squeeze()

Randomly from the entire set

#Displaying images randomly

#Generate random indices to select rows x columns images
rows = 4
cols = 5

ix_img = np.random.choice(len(label_array), size=rows*cols, replace=False)

# Select the images and labels
images = img_array[ix_img]
labels = label_array[ix_img].tolist()

# Define the size of the figure
fig = plt.figure(figsize=(12, 12))

# Create a rows x cols grid of subplots
for i in range(rows*cols):
    ax = fig.add_subplot(rows, cols, i+1)
    ax.imshow(images[i])
    ax.set_xlabel(str(labels[i]), fontsize=10)
    ax.set_xticks([])
    ax.set_yticks([])

# Show the figure
plt.tight_layout()
plt.subplots_adjust(wspace=0.2)
plt.show()

Randomly for each class

#Displaying images (randomly) from each category

num_images = 5

# Create a dictionary to store the indices of each category
category_indices = {}
for category in range(num_classes):
    category_indices[category] = label_df[label_df['Label'] == category].index.tolist()

# Loop through each unique label
for label in name_classes:

    # Get the indices of all images with the current label
    indices = np.where(label_df['Label'] == label)[0]

    # Choose num_images random indices
    random_indices = np.random.choice(indices, size=num_images, replace=False)

    # Get the corresponding images from "img_array"
    images = img_array[random_indices]

    # Create a plot of the images
    fig, ax = plt.subplots(1, num_images, figsize=(20, 4))
    for i, image in enumerate(images):
        ax[i].imshow(image)
        ax[i].axis('off')

    # Set the suptitle and adjust the spacing
    fig.suptitle(label, fontsize=16)
    fig.subplots_adjust(top=1.1, bottom=0.1)
    plt.show()

Output hidden; open in https://colab.research.google.com to view.

Observations from EDA

1. There are 12 unique classes.
2. There are 4,750 records.
3. The classes are imbalanced as seen in the distribution plot.
4. The class "Loose Silky-bent" is rougly three times larger than the class "Maize" or "Common wheat"
5. The images size is 128 x 128 pixes, with 3 channels (RGB).
6. There are images that are out of focus.
7. Some images are dark.
8. Some images contain bar codes, stones, etc.
9. Some images also have random variations that may be due to camera sensor noise, compression artifacts, image processing, user error, etc.
10. Most seedlings are green against a blueshish rock-soil which do not contrast well with the dark green color of most seedlings.

From the dataset image analysis we foresee the need to remove noise from the images. To do this we can use Gaussian Blurring.

Remove Noise using Gaussian Blurring

We will perform Gaussian Blurring to smooth the images by averaging the intensity of each pixel with the intensity of neighboring pixels.

# Choose a random index for the image
index = np.random.randint(0, img_array.shape[0])

# Extract the image and its label
image = img_array[index, :, :, :]
label = label_df.iloc[index, 0]

fig = plt.figure(figsize=(18, 18))

# Plot the original image
plt.subplot(1, 2, 1)
plt.imshow(image.squeeze(), cmap='gray')
plt.title('Original Image (Label: {})'.format(label))
plt.axis('off')

(-0.5, 127.5, 127.5, -0.5)

# Define the kernel size for the Gaussian blur
kernel_size = (3, 3)

# Apply Gaussian blur to each image in the img_array
for i in range(img_array.shape[0]):
    img = img_array[i,:,:,:]
    img = cv2.GaussianBlur(img, kernel_size, 0)
    img_array[i,:,:,:] = img

Apply Normalization

After Gaussian Blurring we will scale the pixel values to a fixed range (i.e. 0 to 1).

# Convert the data type of the img_array to float32
img_array = img_array.astype('float32')

# Normalize the pixel values to the range of 0 to 1
img_array = img_array / 255.0

# Plot the preprocessed image

fig = plt.figure(figsize=(18, 18))
plt.subplot(1, 2, 2)
plt.imshow(image.squeeze(), cmap='gray')
plt.title('Preprocessed Image (Label: {})'.format(label))
plt.axis('off')

# Show the plots
plt.show()

Images after Gaussian Blurring and Normalization

#Displaying images (randomly) from each category

num_images = 5

# Create a dictionary to store the indices of each category
category_indices = {}
for category in range(num_classes):
    category_indices[category] = label_df[label_df['Label'] == category].index.tolist()

# Loop through each unique label
for label in name_classes:

    # Get the indices of all images with the current label
    indices = np.where(label_df['Label'] == label)[0]

    # Choose num_images random indices
    random_indices = np.random.choice(indices, size=num_images, replace=False)

    # Get the corresponding images from "img_array"
    images = img_array[random_indices]

    # Create a plot of the images
    fig, ax = plt.subplots(1, num_images, figsize=(20, 4))
    for i, image in enumerate(images):
        ax[i].imshow(image)
        ax[i].axis('off')

    # Set the suptitle and adjust the spacing
    fig.suptitle(label, fontsize=16)
    fig.subplots_adjust(top=1.1, bottom=0.1)
    plt.show()

Output hidden; open in https://colab.research.google.com to view.

Beware that the differences in the appearance of the original image and the Gaussian-Normalized image are hard to notice by a human eye.

Define a Dictionary to map labels to integer values

# Define a dictionary to map each label to a unique integer value

label_map = {'Small-flowered Cranesbill':0, 'Fat Hen':1, 'Shepherds Purse':2,
       'Common wheat':3, 'Common Chickweed':4, 'Charlock':5, 'Cleavers':6,
       'Scentless Mayweed':7, 'Sugar beet':8, 'Maize':9, 'Black-grass':10,
       'Loose Silky-bent':11}

# Convert the label strings to integer values using the label map
label_df['Label'] = label_df['Label'].map(label_map)

Splitting Data

#splitting data into training and testing set
X_train, X_test, y_train, y_test = train_test_split(img_array,label_df['Label'], test_size=0.1, random_state=42, stratify=label_df['Label'])

Treating Class Imbalance

We can oversample the minority class by adding more samples to that minority classes doing augmentation, generating synthetic data, etc. We can undersample the majority class by removing samples from the majority class to balance the distribution. However we will lose information and although the distribution of the majority class to minitory class is 3 to 1, this may not be a good approach because our dataset is not very large. We can also use data augmentation. And we can also increase the weights of the minority classes.

The data is imbalanced as we have seen earlier. To avoid bias toward the majority class, or underfitting the minority class, we should balance the data.

Balancing the data on the Training set by adjusting weights.

The main reason to use this technique is because reprocessing the images have a very high computational cost.

# Calculate class weights to balance the data
class_weights = {}
for i in range(12):
    class_weights[i] = np.sum(y_train == i) / len(y_train)

print(class_weights)

{0: 0.10432748538011696, 1: 0.09988304093567252, 2: 0.048654970760233916, 3: 0.04654970760233918, 4: 0.1286549707602339, 5: 0.08210526315789474, 6: 0.060350877192982454, 7: 0.10853801169590643, 8: 0.08116959064327485, 9: 0.04654970760233918, 10: 0.05543859649122807, 11: 0.13777777777777778}

# Use the class weights to balance the data
sample_weights = compute_sample_weight(class_weight=class_weights, y=y_train)

Make Data Compatible - One hot Encoding

y_train.info

<bound method Series.info of 437      0
1571     4
838      1
4040    10
4707    11
        ..
746      1
1217     3
3058     7
4682    11
787      1
Name: Label, Length: 4275, dtype: int64>

# Convert the labels to one-hot vector (we only encode the y_train and y_test because these are categorical)
y_train_onehot = to_categorical(y_train, num_classes=12)
y_test_onehot = to_categorical(y_test, num_classes=12)

print("X_train shape:",X_train.shape)
print("X_test shape:",X_test.shape)

print("y_train shape:",y_train.shape)
print("y_test shape:",y_test.shape)

print("y_train one-hot encoded shape:",y_train_onehot.shape)
print("y_test one-hot encoded shape:",y_test_onehot.shape)

X_train shape: (4275, 128, 128, 3)
X_test shape: (475, 128, 128, 3)
y_train shape: (4275,)
y_test shape: (475,)
y_train one-hot encoded shape: (4275, 12)
y_test one-hot encoded shape: (475, 12)

The data is compatible with Keras. img_array is a 4D array with shape (4750, 128, 128, 3), with 4750 images, sized 128x128px and the 3 RGB channels.

We already one hot encoded y_train and y_test, so these are also compatible with Keras.

Convolutional Neural Network Models

input_shape = (128,128,3)

#create dataframe to store scores
Scoresdf = pd.DataFrame(columns =['model', 'accuracy', 'layers', 'total parameters', 'neurons', 'optimizer'])

We will first start with a simple model of only 1 Convolution layer.

Model 0

Model Definition

Layers : 4

Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu

Layer 2: MaxPooling2D layer

Layer 3: Flatten layer

Layer 4: Dense layer : 12 neurons, Relu

Optimizer: Adam

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

# Define the model architecture
model0 = Sequential()

#Convolution with 32 filters size 3,3 and relu
model0.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))

#add max pooling to reduce the size of output of first convolution layer
model0.add(MaxPooling2D(pool_size=(2, 2)))

#reshape the output to a flat vector
model0.add(Flatten())

#add output layer
model0.add(Dense(num_classes, activation='softmax'))

# Compile the model
model0.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# Print the model summary
model0.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 126, 126, 32)      896       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 63, 63, 32)       0         
 )                                                               
                                                                 
 flatten (Flatten)           (None, 127008)            0         
                                                                 
 dense (Dense)               (None, 12)                1524108   
                                                                 
=================================================================
Total params: 1,525,004
Trainable params: 1,525,004
Non-trainable params: 0
_________________________________________________________________

Model Visualization

#visualize the CNN architecture
plot_model(model0, to_file='model0_plot.png', show_shapes=True, show_layer_names=True)

#visualize the CNN architecture using visualkeras
visualkeras.layered_view(model0, legend=True)

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 30 epochs and validation_split as 10%
history0 = model0.fit(X_train, y_train_onehot, epochs=30, batch_size=32,validation_split=0.10,callbacks=[es, mc])

Epoch 1/30
120/121 [============================>.] - ETA: 0s - loss: 1.7592 - accuracy: 0.4187
Epoch 1: val_accuracy improved from -inf to 0.49533, saving model to best_model.h5
121/121 [==============================] - 5s 22ms/step - loss: 1.7579 - accuracy: 0.4195 - val_loss: 1.4393 - val_accuracy: 0.4953
Epoch 2/30
117/121 [============================>.] - ETA: 0s - loss: 1.1295 - accuracy: 0.6242
Epoch 2: val_accuracy improved from 0.49533 to 0.52804, saving model to best_model.h5
121/121 [==============================] - 2s 18ms/step - loss: 1.1295 - accuracy: 0.6249 - val_loss: 1.3444 - val_accuracy: 0.5280
Epoch 3/30
118/121 [============================>.] - ETA: 0s - loss: 0.8919 - accuracy: 0.7076
Epoch 3: val_accuracy improved from 0.52804 to 0.56075, saving model to best_model.h5
121/121 [==============================] - 2s 13ms/step - loss: 0.8918 - accuracy: 0.7065 - val_loss: 1.3244 - val_accuracy: 0.5607
Epoch 4/30
117/121 [============================>.] - ETA: 0s - loss: 0.7481 - accuracy: 0.7591
Epoch 4: val_accuracy improved from 0.56075 to 0.61449, saving model to best_model.h5
121/121 [==============================] - 2s 13ms/step - loss: 0.7480 - accuracy: 0.7593 - val_loss: 1.1457 - val_accuracy: 0.6145
Epoch 5/30
119/121 [============================>.] - ETA: 0s - loss: 0.6163 - accuracy: 0.8120
Epoch 5: val_accuracy improved from 0.61449 to 0.65421, saving model to best_model.h5
121/121 [==============================] - 2s 14ms/step - loss: 0.6152 - accuracy: 0.8123 - val_loss: 1.1489 - val_accuracy: 0.6542
Epoch 6/30
118/121 [============================>.] - ETA: 0s - loss: 0.4893 - accuracy: 0.8620
Epoch 6: val_accuracy improved from 0.65421 to 0.65654, saving model to best_model.h5
121/121 [==============================] - 1s 12ms/step - loss: 0.4904 - accuracy: 0.8617 - val_loss: 1.1100 - val_accuracy: 0.6565
Epoch 7/30
117/121 [============================>.] - ETA: 0s - loss: 0.3744 - accuracy: 0.8993
Epoch 7: val_accuracy did not improve from 0.65654
121/121 [==============================] - 1s 11ms/step - loss: 0.3717 - accuracy: 0.8999 - val_loss: 1.2097 - val_accuracy: 0.6262
Epoch 8/30
117/121 [============================>.] - ETA: 0s - loss: 0.3120 - accuracy: 0.9241
Epoch 8: val_accuracy improved from 0.65654 to 0.66121, saving model to best_model.h5
121/121 [==============================] - 1s 12ms/step - loss: 0.3107 - accuracy: 0.9244 - val_loss: 1.1094 - val_accuracy: 0.6612
Epoch 9/30
119/121 [============================>.] - ETA: 0s - loss: 0.2301 - accuracy: 0.9485
Epoch 9: val_accuracy did not improve from 0.66121
121/121 [==============================] - 1s 11ms/step - loss: 0.2295 - accuracy: 0.9488 - val_loss: 1.2245 - val_accuracy: 0.6589
Epoch 10/30
117/121 [============================>.] - ETA: 0s - loss: 0.1703 - accuracy: 0.9682
Epoch 10: val_accuracy did not improve from 0.66121
121/121 [==============================] - 1s 12ms/step - loss: 0.1722 - accuracy: 0.9670 - val_loss: 1.2595 - val_accuracy: 0.6075
Epoch 11/30
118/121 [============================>.] - ETA: 0s - loss: 0.1336 - accuracy: 0.9783
Epoch 11: val_accuracy did not improve from 0.66121
121/121 [==============================] - 1s 11ms/step - loss: 0.1342 - accuracy: 0.9782 - val_loss: 1.2517 - val_accuracy: 0.6425
Epoch 12/30
117/121 [============================>.] - ETA: 0s - loss: 0.0929 - accuracy: 0.9904
Epoch 12: val_accuracy did not improve from 0.66121
121/121 [==============================] - 1s 12ms/step - loss: 0.0926 - accuracy: 0.9906 - val_loss: 1.3501 - val_accuracy: 0.6145
Epoch 13/30
119/121 [============================>.] - ETA: 0s - loss: 0.0705 - accuracy: 0.9937
Epoch 13: val_accuracy did not improve from 0.66121
121/121 [==============================] - 1s 12ms/step - loss: 0.0708 - accuracy: 0.9938 - val_loss: 1.2872 - val_accuracy: 0.6495
Epoch 14/30
116/121 [===========================>..] - ETA: 0s - loss: 0.0489 - accuracy: 0.9976
Epoch 14: val_accuracy did not improve from 0.66121
121/121 [==============================] - 2s 13ms/step - loss: 0.0483 - accuracy: 0.9977 - val_loss: 1.3812 - val_accuracy: 0.6379
Epoch 15/30
116/121 [===========================>..] - ETA: 0s - loss: 0.0337 - accuracy: 0.9992
Epoch 15: val_accuracy did not improve from 0.66121
121/121 [==============================] - 2s 13ms/step - loss: 0.0335 - accuracy: 0.9992 - val_loss: 1.3267 - val_accuracy: 0.6449
Epoch 16/30
118/121 [============================>.] - ETA: 0s - loss: 0.0285 - accuracy: 0.9981
Epoch 16: val_accuracy did not improve from 0.66121
121/121 [==============================] - 1s 12ms/step - loss: 0.0285 - accuracy: 0.9982 - val_loss: 1.3954 - val_accuracy: 0.6519
Epoch 17/30
120/121 [============================>.] - ETA: 0s - loss: 0.0229 - accuracy: 1.0000
Epoch 17: val_accuracy did not improve from 0.66121
121/121 [==============================] - 1s 12ms/step - loss: 0.0229 - accuracy: 1.0000 - val_loss: 1.4152 - val_accuracy: 0.6449
Epoch 18/30
121/121 [==============================] - ETA: 0s - loss: 0.0156 - accuracy: 1.0000
Epoch 18: val_accuracy did not improve from 0.66121
121/121 [==============================] - 1s 12ms/step - loss: 0.0156 - accuracy: 1.0000 - val_loss: 1.4935 - val_accuracy: 0.6379
Epoch 18: early stopping

Notice that at Epoch 16 the training stopped since the validation loss had continued to increase during 10 epochs. The highest validation accuracy was 0.65654 and validation loss 1.2791.

Model0 - Accuracy vs Epoch curve

plt.plot(history0.history['accuracy'])
plt.plot(history0.history['val_accuracy'])
plt.title('model0 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

We can see that this model performs poorly. The model performs very well on training but very poorly in the validation. It is clearly overfitting.

Evaluation of model0 model.evaluate using the Test data

# Evaluate the model on the test data
score0 = model0.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score0[0])
print('Test accuracy:', score0[1])

Test loss: 1.6907126903533936
Test accuracy: 0.648421049118042

# Test Prediction
y_pred = model0.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)

15/15 [==============================] - 0s 6ms/step

# Test Accuracy
accuracyScore0 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore0)

0.6484210526315789

Classification Report

# Compute the classification report
cr0 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

print('Classification report:\n',cr0)

Classification report:
                            precision    recall  f1-score   support

Small-flowered Cranesbill       0.76      0.84      0.80        50
                  Fat Hen       0.61      0.62      0.62        48
          Shepherds Purse       0.42      0.22      0.29        23
             Common wheat       0.44      0.32      0.37        22
         Common Chickweed       0.93      0.67      0.78        61
                 Charlock       0.83      0.97      0.89        39
                 Cleavers       0.68      0.52      0.59        29
        Scentless Mayweed       0.52      0.81      0.63        52
               Sugar beet       0.64      0.47      0.55        38
                    Maize       0.59      0.59      0.59        22
              Black-grass       0.36      0.38      0.37        26
         Loose Silky-bent       0.65      0.72      0.69        65

                 accuracy                           0.65       475
                macro avg       0.62      0.60      0.60       475
             weighted avg       0.66      0.65      0.64       475

Charlock has a higher precision, recall, and F1 score (0.88, 90, 0.89) than all other classes. Black-grass has the opposite (all low: 0.36, 0.46, 0.41).

Model0 has an overall accuracy of 0.65, it correctly predicted the seedling's class for 65% of the images in the test set.

The precision, recall, and F1-score for each class vary, with some classes having good performance (e.g., Charlock) and others having poor performance (e.g., Shepherds Purse).

The macro average F1 score is 0.65, which means that Model0 is not so good overall. The weighted average F1 score is also .65 and this means that the model is not biased towards any particular class.

Scores_Models_df = Scoresdf.append({'model':'Model0','accuracy':accuracyScore0,'layers':'4','total parameters':'1,525,004', 'neurons':'12', 'optimizer':'Adam'},ignore_index=True)

Confusion Matrix

# Set the confusion matrix
cm0 = confusion_matrix(normal_y_test, y_test_pred_classes)

plt.figure(figsize=(8,6))

sns.heatmap(cm0, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()

Model0 is able to predict some classes with good accuracy but others are not predicted well. We can build better models to improve the performance on the other classes.

This model trained 1,525,004 parameters.

We called our first model, Model0 because this was a simple model with only one Convolution layer, so we didn't expect much from it.

Model 1

Model Definition

This model builds on Model0. Model1 adds a second Convolutional layer and the fully connected layer has 128 neurons to improve the previous Model (Model0).

• Layers : 8
• Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu
• Layer 2: MaxPooling2D layer
• Layer 3: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
• Layer 4: MaxPooling2D layer
• Layer 5: Flatten layer
• Layer 6: Dense layer : 128 neurons, Relu

• Layer 7: Droput layer: 0.5

• Optimizer: Adam

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

Model Implementation

# Define the model architecture
model1 = Sequential()

#Convolution with 32 filters size 3,3 and relu
model1.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))

#add max pooling to reduce the size of output of first convolution layer
model1.add(MaxPooling2D(pool_size=(2, 2)))

#Convolution with 64 filters size 3,3 and relu
model1.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))

#add max pooling to reduce the size of output of second convolution layer
model1.add(MaxPooling2D(pool_size=(2, 2)))

#reshape the output to a flat vector
model1.add(Flatten())

#fully connected layer with 128 neurons
model1.add(Dense(128, activation='relu'))

#add dropout to prevent overfitting
model1.add(Dropout(0.5))

#add output layer
model1.add(Dense(num_classes, activation='softmax'))

# Compile the model
model1.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# Print the model summary
model1.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 126, 126, 32)      896       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 63, 63, 32)       0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 61, 61, 64)        18496     
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 30, 30, 64)       0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 57600)             0         
                                                                 
 dense (Dense)               (None, 128)               7372928   
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_1 (Dense)             (None, 12)                1548      
                                                                 
=================================================================
Total params: 7,393,868
Trainable params: 7,393,868
Non-trainable params: 0
_________________________________________________________________

Model Visualization

#visualize the CNN architecture
plot_model(model1, to_file='model1_plot.png', show_shapes=True, show_layer_names=True)

#visualize the CNN architecture using visualkeras
visualkeras.layered_view(model1, legend=True)

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 60 epochs and validation_split as 10%
history1 = model1.fit(X_train,
          y_train_onehot,
          epochs=60,
          batch_size=32,validation_split=0.10,callbacks=[es, mc])

Epoch 1/60
119/121 [============================>.] - ETA: 0s - loss: 1.9557 - accuracy: 0.3456
Epoch 1: val_accuracy improved from -inf to 0.39953, saving model to best_model.h5
121/121 [==============================] - 4s 24ms/step - loss: 1.9516 - accuracy: 0.3473 - val_loss: 1.6871 - val_accuracy: 0.3995
Epoch 2/60
118/121 [============================>.] - ETA: 0s - loss: 1.4500 - accuracy: 0.5085
Epoch 2: val_accuracy improved from 0.39953 to 0.57009, saving model to best_model.h5
121/121 [==============================] - 3s 24ms/step - loss: 1.4500 - accuracy: 0.5095 - val_loss: 1.2356 - val_accuracy: 0.5701
Epoch 3/60
118/121 [============================>.] - ETA: 0s - loss: 1.2128 - accuracy: 0.5879
Epoch 3: val_accuracy improved from 0.57009 to 0.59813, saving model to best_model.h5
121/121 [==============================] - 3s 24ms/step - loss: 1.2120 - accuracy: 0.5885 - val_loss: 1.2166 - val_accuracy: 0.5981
Epoch 4/60
118/121 [============================>.] - ETA: 0s - loss: 1.0462 - accuracy: 0.6496
Epoch 4: val_accuracy improved from 0.59813 to 0.63785, saving model to best_model.h5
121/121 [==============================] - 3s 23ms/step - loss: 1.0478 - accuracy: 0.6483 - val_loss: 1.0640 - val_accuracy: 0.6379
Epoch 5/60
118/121 [============================>.] - ETA: 0s - loss: 0.9670 - accuracy: 0.6700
Epoch 5: val_accuracy improved from 0.63785 to 0.66121, saving model to best_model.h5
121/121 [==============================] - 2s 21ms/step - loss: 0.9692 - accuracy: 0.6691 - val_loss: 1.0265 - val_accuracy: 0.6612
Epoch 6/60
121/121 [==============================] - ETA: 0s - loss: 0.8473 - accuracy: 0.7024
Epoch 6: val_accuracy improved from 0.66121 to 0.67523, saving model to best_model.h5
121/121 [==============================] - 5s 39ms/step - loss: 0.8473 - accuracy: 0.7024 - val_loss: 0.9550 - val_accuracy: 0.6752
Epoch 7/60
120/121 [============================>.] - ETA: 0s - loss: 0.7730 - accuracy: 0.7372
Epoch 7: val_accuracy did not improve from 0.67523
121/121 [==============================] - 2s 20ms/step - loss: 0.7724 - accuracy: 0.7375 - val_loss: 0.9919 - val_accuracy: 0.6589
Epoch 8/60
120/121 [============================>.] - ETA: 0s - loss: 0.6773 - accuracy: 0.7633
Epoch 8: val_accuracy improved from 0.67523 to 0.71729, saving model to best_model.h5
121/121 [==============================] - 3s 22ms/step - loss: 0.6779 - accuracy: 0.7627 - val_loss: 0.9073 - val_accuracy: 0.7173
Epoch 9/60
120/121 [============================>.] - ETA: 0s - loss: 0.5863 - accuracy: 0.7974
Epoch 9: val_accuracy did not improve from 0.71729
121/121 [==============================] - 2s 18ms/step - loss: 0.5861 - accuracy: 0.7975 - val_loss: 0.9788 - val_accuracy: 0.7033
Epoch 10/60
119/121 [============================>.] - ETA: 0s - loss: 0.5086 - accuracy: 0.8262
Epoch 10: val_accuracy improved from 0.71729 to 0.72196, saving model to best_model.h5
121/121 [==============================] - 3s 21ms/step - loss: 0.5104 - accuracy: 0.8251 - val_loss: 0.9273 - val_accuracy: 0.7220
Epoch 11/60
119/121 [============================>.] - ETA: 0s - loss: 0.4790 - accuracy: 0.8290
Epoch 11: val_accuracy improved from 0.72196 to 0.72664, saving model to best_model.h5
121/121 [==============================] - 2s 20ms/step - loss: 0.4803 - accuracy: 0.8290 - val_loss: 0.9444 - val_accuracy: 0.7266
Epoch 12/60
121/121 [==============================] - ETA: 0s - loss: 0.4102 - accuracy: 0.8568
Epoch 12: val_accuracy improved from 0.72664 to 0.73131, saving model to best_model.h5
121/121 [==============================] - 3s 22ms/step - loss: 0.4102 - accuracy: 0.8568 - val_loss: 0.9420 - val_accuracy: 0.7313
Epoch 13/60
118/121 [============================>.] - ETA: 0s - loss: 0.3798 - accuracy: 0.8673
Epoch 13: val_accuracy improved from 0.73131 to 0.74766, saving model to best_model.h5
121/121 [==============================] - 3s 23ms/step - loss: 0.3841 - accuracy: 0.8661 - val_loss: 0.9019 - val_accuracy: 0.7477
Epoch 14/60
121/121 [==============================] - ETA: 0s - loss: 0.3665 - accuracy: 0.8742
Epoch 14: val_accuracy improved from 0.74766 to 0.75234, saving model to best_model.h5
121/121 [==============================] - 3s 22ms/step - loss: 0.3665 - accuracy: 0.8742 - val_loss: 0.9895 - val_accuracy: 0.7523
Epoch 15/60
118/121 [============================>.] - ETA: 0s - loss: 0.3009 - accuracy: 0.8938
Epoch 15: val_accuracy did not improve from 0.75234
121/121 [==============================] - 2s 18ms/step - loss: 0.2996 - accuracy: 0.8945 - val_loss: 0.9900 - val_accuracy: 0.7407
Epoch 16/60
121/121 [==============================] - ETA: 0s - loss: 0.2900 - accuracy: 0.8997
Epoch 16: val_accuracy did not improve from 0.75234
121/121 [==============================] - 3s 21ms/step - loss: 0.2900 - accuracy: 0.8997 - val_loss: 0.9611 - val_accuracy: 0.7407
Epoch 17/60
120/121 [============================>.] - ETA: 0s - loss: 0.2824 - accuracy: 0.8979
Epoch 17: val_accuracy did not improve from 0.75234
121/121 [==============================] - 2s 18ms/step - loss: 0.2824 - accuracy: 0.8978 - val_loss: 1.0341 - val_accuracy: 0.7407
Epoch 18/60
120/121 [============================>.] - ETA: 0s - loss: 0.2479 - accuracy: 0.9146
Epoch 18: val_accuracy did not improve from 0.75234
121/121 [==============================] - 2s 19ms/step - loss: 0.2483 - accuracy: 0.9145 - val_loss: 1.1104 - val_accuracy: 0.7453
Epoch 19/60
120/121 [============================>.] - ETA: 0s - loss: 0.2387 - accuracy: 0.9128
Epoch 19: val_accuracy did not improve from 0.75234
121/121 [==============================] - 2s 20ms/step - loss: 0.2383 - accuracy: 0.9129 - val_loss: 1.0547 - val_accuracy: 0.7477
Epoch 20/60
118/121 [============================>.] - ETA: 0s - loss: 0.2062 - accuracy: 0.9266
Epoch 20: val_accuracy did not improve from 0.75234
121/121 [==============================] - 2s 20ms/step - loss: 0.2065 - accuracy: 0.9264 - val_loss: 1.0478 - val_accuracy: 0.7523
Epoch 21/60
119/121 [============================>.] - ETA: 0s - loss: 0.2306 - accuracy: 0.9160
Epoch 21: val_accuracy did not improve from 0.75234
121/121 [==============================] - 2s 18ms/step - loss: 0.2302 - accuracy: 0.9163 - val_loss: 1.2161 - val_accuracy: 0.7407
Epoch 22/60
119/121 [============================>.] - ETA: 0s - loss: 0.2000 - accuracy: 0.9280
Epoch 22: val_accuracy did not improve from 0.75234
121/121 [==============================] - 2s 18ms/step - loss: 0.2001 - accuracy: 0.9280 - val_loss: 1.1027 - val_accuracy: 0.7430
Epoch 23/60
118/121 [============================>.] - ETA: 0s - loss: 0.1923 - accuracy: 0.9301
Epoch 23: val_accuracy did not improve from 0.75234
121/121 [==============================] - 2s 18ms/step - loss: 0.1908 - accuracy: 0.9309 - val_loss: 1.1310 - val_accuracy: 0.7383
Epoch 23: early stopping

Notice that at Epoch 23 the training stopped since the validation loss had continued to increase during 10 epochs. The highest validation accuracy was 0.75 and validation loss of 1.1866.

Model1 - Accuracy vs Epoch curve

plt.plot(history1.history['accuracy'])
plt.plot(history1.history['val_accuracy'])
plt.title('model1 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

Notice that the model shows overfitting. Past epoch 12 the validation loss doesn't reduce or begins to increase.

The model performs poorly as we see that the training accuracy is high but the validation accuracy is poor, indicating that the model is overfitting.

Evaluation of model1 model.evaluate using the Test data

# Evaluate the model on the test data
score1 = model1.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score1[0])
print('Test accuracy:', score1[1])

Test loss: 1.2217235565185547
Test accuracy: 0.730526328086853

# Test Prediction
y_pred = model1.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)

15/15 [==============================] - 0s 7ms/step

# Test Accuracy
accuracyScore1 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore1)

0.7305263157894737

Classification Report

# Compute the classification report
cr1 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

print('Classification report:\n',cr1)

Classification report:
                            precision    recall  f1-score   support

Small-flowered Cranesbill       0.90      0.86      0.88        50
                  Fat Hen       0.75      0.75      0.75        48
          Shepherds Purse       0.43      0.39      0.41        23
             Common wheat       0.65      0.68      0.67        22
         Common Chickweed       0.88      0.84      0.86        61
                 Charlock       0.77      0.92      0.84        39
                 Cleavers       0.83      0.69      0.75        29
        Scentless Mayweed       0.64      0.75      0.69        52
               Sugar beet       0.80      0.63      0.71        38
                    Maize       0.57      0.73      0.64        22
              Black-grass       0.42      0.31      0.36        26
         Loose Silky-bent       0.74      0.77      0.75        65

                 accuracy                           0.73       475
                macro avg       0.70      0.69      0.69       475
             weighted avg       0.73      0.73      0.73       475

The model seems to perform well for classes such as Small-flowered Cranesbill, Common Chickweed, and Charlock with high precision, recall, and F1-score.

The model performs poorly with classes like Black-grass, Shepherds Purse, and Common wheat that have low precision, recall, and F1-score.

The macro-average and weighted-average F1-score are 0.69 and 0.73, which indicates that the model has an overall decent performance.

However, this model can be improved, especially for the clasess that perform poorly.

Scoresdf = Scoresdf.append({'model':'Model1','accuracy':accuracyScore1,'layers':'8','total parameters':'7,393,868', 'neurons':'128', 'optimizer':'Adam'},ignore_index=True)

Confusion Matrix

# Set the confusion matrix
cm1 = confusion_matrix(normal_y_test, y_test_pred_classes)

plt.figure(figsize=(8,6))

sns.heatmap(cm1, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()

Model1 one performs better than Model0 but the model isn't very good either. It trains 7,393,868 parameters which is about 4 times more parameters than Model0 and although this difference is massive, Model1 doesn't have a great performance either.

Model 2

Model2 keeps growing, to try to improve on the previous model (Model1) we added more layers.

• Layers : 10

• Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu

• Layer 2: MaxPooling2D layer
• Layer 3: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
• Layer 4: MaxPooling2D layer
• Layer 5: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
• Layer 6: MaxPooling2D layer
• Layer 7: Flatten layer
• Layer 8: Dense layer : 128 neurons, Relu

• Layer 9: Droput layer: 0.5

• Optimizer: Adam

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

Model Implementation

# Define the model architecture
model2 = Sequential()

#Convolution with 32 filters size 3,3 and relu
model2.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=(128,128,3)))

#add max pooling to reduce the size of output of first convolution layer
model2.add(MaxPooling2D(pool_size=(2,2)))

#Convolution with 64 filters size 3,3 and relu
model2.add(Conv2D(64, kernel_size=(3,3), activation='relu'))

#add max pooling to reduce the size of output of first convolution layer
model2.add(MaxPooling2D(pool_size=(2,2)))

#Convolution with 128 filters size 3,3 and relu
model2.add(Conv2D(128, kernel_size=(3,3), activation='relu'))

#add max pooling to reduce the size of output of first convolution layer
model2.add(MaxPooling2D(pool_size=(2,2)))

#reshape the output to a flat vector
model2.add(Flatten())

#fully connected layer with 128 neurons
model2.add(Dense(128, activation='relu'))

#add dropout to prevent overfitting
model2.add(Dropout(0.5))

#add output layer
model2.add(Dense(num_classes, activation='softmax'))

# Compile the model
model2.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# Print the model summary
model2.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 126, 126, 32)      896       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 63, 63, 32)       0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 61, 61, 64)        18496     
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 30, 30, 64)       0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 28, 28, 128)       73856     
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 14, 14, 128)      0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 25088)             0         
                                                                 
 dense (Dense)               (None, 128)               3211392   
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_1 (Dense)             (None, 12)                1548      
                                                                 
=================================================================
Total params: 3,306,188
Trainable params: 3,306,188
Non-trainable params: 0
_________________________________________________________________

Model Visualization

#visualize the cnn model architecture
plot_model(model2, to_file='model2_plot.png', show_shapes=True, show_layer_names=True)

#visualize the cnn model architecture using visualkeras
visualkeras.layered_view(model2, legend=True)

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 40 epochs and validation_split as 10%
history2 = model2.fit(X_train,
          y_train_onehot,
          epochs=40,
          batch_size=32,validation_split=0.10,callbacks=[es, mc])

Epoch 1/40
119/121 [============================>.] - ETA: 0s - loss: 1.9732 - accuracy: 0.3369
Epoch 1: val_accuracy improved from -inf to 0.44626, saving model to best_model.h5
121/121 [==============================] - 6s 25ms/step - loss: 1.9721 - accuracy: 0.3379 - val_loss: 1.5570 - val_accuracy: 0.4463
Epoch 2/40
118/121 [============================>.] - ETA: 0s - loss: 1.4752 - accuracy: 0.4923
Epoch 2: val_accuracy improved from 0.44626 to 0.52570, saving model to best_model.h5
121/121 [==============================] - 3s 21ms/step - loss: 1.4720 - accuracy: 0.4939 - val_loss: 1.2610 - val_accuracy: 0.5257
Epoch 3/40
118/121 [============================>.] - ETA: 0s - loss: 1.2448 - accuracy: 0.5728
Epoch 3: val_accuracy improved from 0.52570 to 0.61916, saving model to best_model.h5
121/121 [==============================] - 3s 21ms/step - loss: 1.2429 - accuracy: 0.5734 - val_loss: 1.0807 - val_accuracy: 0.6192
Epoch 4/40
118/121 [============================>.] - ETA: 0s - loss: 1.0740 - accuracy: 0.6382
Epoch 4: val_accuracy did not improve from 0.61916
121/121 [==============================] - 2s 19ms/step - loss: 1.0733 - accuracy: 0.6382 - val_loss: 1.0088 - val_accuracy: 0.6145
Epoch 5/40
119/121 [============================>.] - ETA: 0s - loss: 0.9800 - accuracy: 0.6594
Epoch 5: val_accuracy improved from 0.61916 to 0.69159, saving model to best_model.h5
121/121 [==============================] - 3s 22ms/step - loss: 0.9808 - accuracy: 0.6584 - val_loss: 0.8978 - val_accuracy: 0.6916
Epoch 6/40
120/121 [============================>.] - ETA: 0s - loss: 0.8985 - accuracy: 0.6867
Epoch 6: val_accuracy improved from 0.69159 to 0.74065, saving model to best_model.h5
121/121 [==============================] - 3s 22ms/step - loss: 0.8982 - accuracy: 0.6868 - val_loss: 0.8329 - val_accuracy: 0.7407
Epoch 7/40
118/121 [============================>.] - ETA: 0s - loss: 0.8048 - accuracy: 0.7259
Epoch 7: val_accuracy did not improve from 0.74065
121/121 [==============================] - 2s 20ms/step - loss: 0.8049 - accuracy: 0.7255 - val_loss: 0.8244 - val_accuracy: 0.7407
Epoch 8/40
118/121 [============================>.] - ETA: 0s - loss: 0.7244 - accuracy: 0.7407
Epoch 8: val_accuracy improved from 0.74065 to 0.76636, saving model to best_model.h5
121/121 [==============================] - 3s 21ms/step - loss: 0.7228 - accuracy: 0.7406 - val_loss: 0.7660 - val_accuracy: 0.7664
Epoch 9/40
118/121 [============================>.] - ETA: 0s - loss: 0.6450 - accuracy: 0.7688
Epoch 9: val_accuracy improved from 0.76636 to 0.77103, saving model to best_model.h5
121/121 [==============================] - 3s 21ms/step - loss: 0.6472 - accuracy: 0.7687 - val_loss: 0.7315 - val_accuracy: 0.7710
Epoch 10/40
118/121 [============================>.] - ETA: 0s - loss: 0.6278 - accuracy: 0.7773
Epoch 10: val_accuracy did not improve from 0.77103
121/121 [==============================] - 2s 20ms/step - loss: 0.6289 - accuracy: 0.7757 - val_loss: 0.7808 - val_accuracy: 0.7407
Epoch 11/40
119/121 [============================>.] - ETA: 0s - loss: 0.5677 - accuracy: 0.7946
Epoch 11: val_accuracy did not improve from 0.77103
121/121 [==============================] - 3s 22ms/step - loss: 0.5674 - accuracy: 0.7952 - val_loss: 0.7369 - val_accuracy: 0.7593
Epoch 12/40
120/121 [============================>.] - ETA: 0s - loss: 0.5108 - accuracy: 0.8104
Epoch 12: val_accuracy improved from 0.77103 to 0.78037, saving model to best_model.h5
121/121 [==============================] - 3s 23ms/step - loss: 0.5105 - accuracy: 0.8105 - val_loss: 0.6926 - val_accuracy: 0.7804
Epoch 13/40
121/121 [==============================] - ETA: 0s - loss: 0.4677 - accuracy: 0.8227
Epoch 13: val_accuracy improved from 0.78037 to 0.80140, saving model to best_model.h5
121/121 [==============================] - 3s 22ms/step - loss: 0.4677 - accuracy: 0.8227 - val_loss: 0.6959 - val_accuracy: 0.8014
Epoch 14/40
118/121 [============================>.] - ETA: 0s - loss: 0.4548 - accuracy: 0.8276
Epoch 14: val_accuracy did not improve from 0.80140
121/121 [==============================] - 2s 20ms/step - loss: 0.4534 - accuracy: 0.8290 - val_loss: 0.7239 - val_accuracy: 0.7897
Epoch 15/40
118/121 [============================>.] - ETA: 0s - loss: 0.4509 - accuracy: 0.8337
Epoch 15: val_accuracy did not improve from 0.80140
121/121 [==============================] - 2s 20ms/step - loss: 0.4492 - accuracy: 0.8349 - val_loss: 0.7702 - val_accuracy: 0.7850
Epoch 16/40
118/121 [============================>.] - ETA: 0s - loss: 0.4024 - accuracy: 0.8490
Epoch 16: val_accuracy did not improve from 0.80140
121/121 [==============================] - 2s 20ms/step - loss: 0.4057 - accuracy: 0.8479 - val_loss: 0.7375 - val_accuracy: 0.8014
Epoch 17/40
121/121 [==============================] - ETA: 0s - loss: 0.3776 - accuracy: 0.8589
Epoch 17: val_accuracy did not improve from 0.80140
121/121 [==============================] - 3s 22ms/step - loss: 0.3776 - accuracy: 0.8589 - val_loss: 0.7692 - val_accuracy: 0.7991
Epoch 18/40
118/121 [============================>.] - ETA: 0s - loss: 0.3786 - accuracy: 0.8594
Epoch 18: val_accuracy did not improve from 0.80140
121/121 [==============================] - 3s 21ms/step - loss: 0.3786 - accuracy: 0.8586 - val_loss: 0.8283 - val_accuracy: 0.7874
Epoch 19/40
120/121 [============================>.] - ETA: 0s - loss: 0.3526 - accuracy: 0.8641
Epoch 19: val_accuracy did not improve from 0.80140
121/121 [==============================] - 3s 21ms/step - loss: 0.3523 - accuracy: 0.8643 - val_loss: 0.7889 - val_accuracy: 0.7944
Epoch 20/40
119/121 [============================>.] - ETA: 0s - loss: 0.3298 - accuracy: 0.8737
Epoch 20: val_accuracy did not improve from 0.80140
121/121 [==============================] - 3s 27ms/step - loss: 0.3314 - accuracy: 0.8734 - val_loss: 0.8751 - val_accuracy: 0.7804
Epoch 21/40
118/121 [============================>.] - ETA: 0s - loss: 0.3306 - accuracy: 0.8718
Epoch 21: val_accuracy improved from 0.80140 to 0.80841, saving model to best_model.h5
121/121 [==============================] - 3s 21ms/step - loss: 0.3314 - accuracy: 0.8711 - val_loss: 0.7938 - val_accuracy: 0.8084
Epoch 22/40
121/121 [==============================] - ETA: 0s - loss: 0.3051 - accuracy: 0.8807
Epoch 22: val_accuracy did not improve from 0.80841
121/121 [==============================] - 2s 20ms/step - loss: 0.3051 - accuracy: 0.8807 - val_loss: 0.7822 - val_accuracy: 0.7921
Epoch 22: early stopping

The training stopped at epoch 21 because the validation loss had increased during 10 epochs. The highest validation accuracy was 0.780 and validation loss 0.7628.

Model 2: Accuracy vs epochs plot

plt.plot(history2.history['accuracy'])
plt.plot(history2.history['val_accuracy'])
plt.title('model2 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

Notice that somewhere around epoch 15 the validation loss begins to increase.

Overall Model2 seems to perform better than the 2 previous models.

Evaluation of model2 model.evaluate using the Test data

# Evaluate the model on the test data
score2 = model2.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score2[0])
print('Test accuracy:', score2[1])

Test loss: 0.9905914068222046
Test accuracy: 0.7873684167861938

# Test Prediction
y_pred = model2.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)

15/15 [==============================] - 0s 7ms/step

# Test Accuracy
accuracyScore2 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore2)

0.7873684210526316

Classification Report

# Compute the classification report
cr2 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

print('Classification report:\n',cr2)

Classification report:
                            precision    recall  f1-score   support

Small-flowered Cranesbill       0.86      0.86      0.86        50
                  Fat Hen       0.75      0.88      0.81        48
          Shepherds Purse       0.78      0.30      0.44        23
             Common wheat       0.73      0.73      0.73        22
         Common Chickweed       0.92      0.95      0.94        61
                 Charlock       0.90      0.92      0.91        39
                 Cleavers       0.81      0.86      0.83        29
        Scentless Mayweed       0.70      0.75      0.72        52
               Sugar beet       0.88      0.76      0.82        38
                    Maize       0.65      0.59      0.62        22
              Black-grass       0.50      0.15      0.24        26
         Loose Silky-bent       0.71      0.95      0.82        65

                 accuracy                           0.79       475
                macro avg       0.76      0.73      0.73       475
             weighted avg       0.78      0.79      0.77       475

The overall accuracy of Model2 is 0.79, which is a ok. The precision and recall values for each class are also mostly good, with some classes having higher values than others.

The highest precision and recall values are achieved for "Small-flowered Cranesbill", "Fat Hen", "Common Chickweed", "Charlock", and "Cleavers", which means that the model can identify well these classes.

Some classes have lower precision and recall values, such as "Shepherds Purse", "Common wheat", "Scentless Mayweed", "Sugar beet", "Maize", and "Black-grass".

Scoresdf = Scoresdf.append({'model':'Model2','accuracy':accuracyScore2,'layers':'10','total parameters':'3,306,188', 'neurons':'128', 'optimizer':'Adam'},ignore_index=True)

Confusion Matrix

#set and plot the Confusion matrix
cm2 = confusion_matrix(normal_y_test, y_test_pred_classes)

plt.figure(figsize=(8,6))
sns.heatmap(cm2, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()

Model2 performs better than Model0 and Model1. It actually performs OK.

But we can improve the model for better performance.

Model 3

To try to improve Model2, we change the optimizer from Adam to SGD.

• Layers : 10

• Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu

• Layer 2: MaxPooling2D layer
• Layer 3: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
• Layer 4: MaxPooling2D layer
• Layer 5: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
• Layer 6: MaxPooling2D layer
• Layer 7: Flatten layer
• Layer 8: Dense layer : 128 neurons, Relu

• Layer 9: Droput layer: 0.5

• Optimizer: SGD

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

Lets used SGD as optimizer with the second model since we obtained a higher accuracy than the first model

# Using SGD Optimizer
sgd = SGD(learning_rate=0.01, momentum=0.9)

Model Implementation

# Define the model architecture
model3 = Sequential()

#Convolution with 32 filters size 3,3 and relu
model3.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=(128,128,3)))

#add max pooling to reduce the size of output of first convolution layer
model3.add(MaxPooling2D(pool_size=(2,2)))

#Convolution with 64 filters size 3,3 and relu
model3.add(Conv2D(64, kernel_size=(3,3), activation='relu'))

#add max pooling to reduce the size of output of first convolution layer
model3.add(MaxPooling2D(pool_size=(2,2)))

#Convolution with 128 filters size 3,3 and relu
model3.add(Conv2D(128, kernel_size=(3,3), activation='relu'))

#add max pooling to reduce the size of output of first convolution layer
model3.add(MaxPooling2D(pool_size=(2,2)))

#reshape the output to a flat vector
model3.add(Flatten())

#fully connected layer with 128 neurons
model3.add(Dense(128, activation='relu'))

#add dropout to prevent overfitting
model3.add(Dropout(0.5))

#add output layer
model3.add(Dense(num_classes, activation='softmax'))

# Compile the model
model3.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

# Print the model summary
model3.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 126, 126, 32)      896       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 63, 63, 32)       0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 61, 61, 64)        18496     
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 30, 30, 64)       0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 28, 28, 128)       73856     
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 14, 14, 128)      0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 25088)             0         
                                                                 
 dense (Dense)               (None, 128)               3211392   
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_1 (Dense)             (None, 12)                1548      
                                                                 
=================================================================
Total params: 3,306,188
Trainable params: 3,306,188
Non-trainable params: 0
_________________________________________________________________

Model Visualization

#visualize the cnn model architecture
plot_model(model3, to_file='model3_plot.png', show_shapes=True, show_layer_names=True)

#visualize the cnn model architecture using visualkeras
visualkeras.layered_view(model3, legend=True)

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 40 epochs and validation_split as 10%
history3 = model3.fit(X_train, y_train_onehot, epochs=40, batch_size=32,validation_split=0.10,callbacks=[es, mc])

Epoch 1/40
121/121 [==============================] - ETA: 0s - loss: 2.3845 - accuracy: 0.1578
Epoch 1: val_accuracy improved from -inf to 0.32243, saving model to best_model.h5
121/121 [==============================] - 4s 26ms/step - loss: 2.3845 - accuracy: 0.1578 - val_loss: 2.0716 - val_accuracy: 0.3224
Epoch 2/40
118/121 [============================>.] - ETA: 0s - loss: 1.8602 - accuracy: 0.3665
Epoch 2: val_accuracy did not improve from 0.32243
121/121 [==============================] - 2s 20ms/step - loss: 1.8570 - accuracy: 0.3678 - val_loss: 2.1210 - val_accuracy: 0.2967
Epoch 3/40
118/121 [============================>.] - ETA: 0s - loss: 1.6527 - accuracy: 0.4256
Epoch 3: val_accuracy improved from 0.32243 to 0.45794, saving model to best_model.h5
121/121 [==============================] - 2s 20ms/step - loss: 1.6484 - accuracy: 0.4273 - val_loss: 1.5749 - val_accuracy: 0.4579
Epoch 4/40
119/121 [============================>.] - ETA: 0s - loss: 1.4911 - accuracy: 0.4811
Epoch 4: val_accuracy did not improve from 0.45794
121/121 [==============================] - 2s 19ms/step - loss: 1.4852 - accuracy: 0.4830 - val_loss: 1.4450 - val_accuracy: 0.4579
Epoch 5/40
118/121 [============================>.] - ETA: 0s - loss: 1.3449 - accuracy: 0.5246
Epoch 5: val_accuracy improved from 0.45794 to 0.46028, saving model to best_model.h5
121/121 [==============================] - 2s 20ms/step - loss: 1.3446 - accuracy: 0.5251 - val_loss: 1.6715 - val_accuracy: 0.4603
Epoch 6/40
118/121 [============================>.] - ETA: 0s - loss: 1.2553 - accuracy: 0.5662
Epoch 6: val_accuracy improved from 0.46028 to 0.50234, saving model to best_model.h5
121/121 [==============================] - 3s 21ms/step - loss: 1.2546 - accuracy: 0.5667 - val_loss: 1.4218 - val_accuracy: 0.5023
Epoch 7/40
118/121 [============================>.] - ETA: 0s - loss: 1.1140 - accuracy: 0.6163
Epoch 7: val_accuracy improved from 0.50234 to 0.62850, saving model to best_model.h5
121/121 [==============================] - 3s 22ms/step - loss: 1.1108 - accuracy: 0.6168 - val_loss: 1.1198 - val_accuracy: 0.6285
Epoch 8/40
120/121 [============================>.] - ETA: 0s - loss: 1.0393 - accuracy: 0.6391
Epoch 8: val_accuracy improved from 0.62850 to 0.63551, saving model to best_model.h5
121/121 [==============================] - 2s 20ms/step - loss: 1.0389 - accuracy: 0.6389 - val_loss: 1.0225 - val_accuracy: 0.6355
Epoch 9/40
119/121 [============================>.] - ETA: 0s - loss: 0.9215 - accuracy: 0.6786
Epoch 9: val_accuracy improved from 0.63551 to 0.69393, saving model to best_model.h5
121/121 [==============================] - 2s 21ms/step - loss: 0.9207 - accuracy: 0.6792 - val_loss: 0.9386 - val_accuracy: 0.6939
Epoch 10/40
118/121 [============================>.] - ETA: 0s - loss: 0.9077 - accuracy: 0.6851
Epoch 10: val_accuracy did not improve from 0.69393
121/121 [==============================] - 2s 20ms/step - loss: 0.9055 - accuracy: 0.6862 - val_loss: 0.9669 - val_accuracy: 0.6799
Epoch 11/40
118/121 [============================>.] - ETA: 0s - loss: 0.7901 - accuracy: 0.7219
Epoch 11: val_accuracy did not improve from 0.69393
121/121 [==============================] - 2s 20ms/step - loss: 0.7900 - accuracy: 0.7219 - val_loss: 1.0805 - val_accuracy: 0.6355
Epoch 12/40
119/121 [============================>.] - ETA: 0s - loss: 0.7174 - accuracy: 0.7453
Epoch 12: val_accuracy improved from 0.69393 to 0.73364, saving model to best_model.h5
121/121 [==============================] - 3s 21ms/step - loss: 0.7178 - accuracy: 0.7447 - val_loss: 0.8993 - val_accuracy: 0.7336
Epoch 13/40
119/121 [============================>.] - ETA: 0s - loss: 0.6675 - accuracy: 0.7655
Epoch 13: val_accuracy improved from 0.73364 to 0.73832, saving model to best_model.h5
121/121 [==============================] - 3s 22ms/step - loss: 0.6708 - accuracy: 0.7640 - val_loss: 0.8316 - val_accuracy: 0.7383
Epoch 14/40
119/121 [============================>.] - ETA: 0s - loss: 0.6881 - accuracy: 0.7595
Epoch 14: val_accuracy did not improve from 0.73832
121/121 [==============================] - 2s 20ms/step - loss: 0.6882 - accuracy: 0.7596 - val_loss: 0.8765 - val_accuracy: 0.7383
Epoch 15/40
118/121 [============================>.] - ETA: 0s - loss: 0.6067 - accuracy: 0.7844
Epoch 15: val_accuracy improved from 0.73832 to 0.79673, saving model to best_model.h5
121/121 [==============================] - 2s 20ms/step - loss: 0.6055 - accuracy: 0.7848 - val_loss: 0.7043 - val_accuracy: 0.7967
Epoch 16/40
118/121 [============================>.] - ETA: 0s - loss: 0.5655 - accuracy: 0.7969
Epoch 16: val_accuracy did not improve from 0.79673
121/121 [==============================] - 2s 20ms/step - loss: 0.5661 - accuracy: 0.7962 - val_loss: 0.8609 - val_accuracy: 0.7500
Epoch 17/40
119/121 [============================>.] - ETA: 0s - loss: 0.4954 - accuracy: 0.8180
Epoch 17: val_accuracy did not improve from 0.79673
121/121 [==============================] - 2s 20ms/step - loss: 0.4939 - accuracy: 0.8186 - val_loss: 0.7640 - val_accuracy: 0.7593
Epoch 18/40
118/121 [============================>.] - ETA: 0s - loss: 0.4738 - accuracy: 0.8244
Epoch 18: val_accuracy did not improve from 0.79673
121/121 [==============================] - 3s 21ms/step - loss: 0.4720 - accuracy: 0.8251 - val_loss: 0.8498 - val_accuracy: 0.7734
Epoch 19/40
118/121 [============================>.] - ETA: 0s - loss: 0.4325 - accuracy: 0.8392
Epoch 19: val_accuracy did not improve from 0.79673
121/121 [==============================] - 3s 21ms/step - loss: 0.4319 - accuracy: 0.8391 - val_loss: 0.7769 - val_accuracy: 0.7710
Epoch 20/40
118/121 [============================>.] - ETA: 0s - loss: 0.4054 - accuracy: 0.8520
Epoch 20: val_accuracy did not improve from 0.79673
121/121 [==============================] - 2s 20ms/step - loss: 0.4040 - accuracy: 0.8531 - val_loss: 0.8047 - val_accuracy: 0.7780
Epoch 21/40
119/121 [============================>.] - ETA: 0s - loss: 0.3600 - accuracy: 0.8640
Epoch 21: val_accuracy improved from 0.79673 to 0.80140, saving model to best_model.h5
121/121 [==============================] - 2s 20ms/step - loss: 0.3609 - accuracy: 0.8648 - val_loss: 0.8184 - val_accuracy: 0.8014
Epoch 22/40
118/121 [============================>.] - ETA: 0s - loss: 0.3771 - accuracy: 0.8604
Epoch 22: val_accuracy did not improve from 0.80140
121/121 [==============================] - 2s 19ms/step - loss: 0.3747 - accuracy: 0.8612 - val_loss: 0.8573 - val_accuracy: 0.7477
Epoch 23/40
118/121 [============================>.] - ETA: 0s - loss: 0.3378 - accuracy: 0.8816
Epoch 23: val_accuracy did not improve from 0.80140
121/121 [==============================] - 2s 20ms/step - loss: 0.3357 - accuracy: 0.8825 - val_loss: 0.8310 - val_accuracy: 0.7640
Epoch 24/40
118/121 [============================>.] - ETA: 0s - loss: 0.3023 - accuracy: 0.8922
Epoch 24: val_accuracy did not improve from 0.80140
121/121 [==============================] - 3s 21ms/step - loss: 0.2997 - accuracy: 0.8932 - val_loss: 0.9448 - val_accuracy: 0.7593
Epoch 25/40
118/121 [============================>.] - ETA: 0s - loss: 0.3030 - accuracy: 0.8856
Epoch 25: val_accuracy did not improve from 0.80140
121/121 [==============================] - 3s 21ms/step - loss: 0.3038 - accuracy: 0.8856 - val_loss: 0.9751 - val_accuracy: 0.7827
Epoch 25: early stopping

Training stopped on epoch 28. The highest validation accuracy was 0.7921 where the validation loss was 0.844.

Model3 Accuracy vs Epoch curve

plt.plot(history3.history['accuracy'])
plt.plot(history3.history['val_accuracy'])
plt.title('model3 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

Around epoch 20, the validation accuracy is highest. The model seems to be performing decently around epoch 22. There is some overfitting in the data.

Evaluation of model3 model.evaluate using the Test data

# Evaluate the model on the test data
score3 = model3.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score3[0])
print('Test accuracy:', score3[1])

Test loss: 0.9461641907691956
Test accuracy: 0.7726315855979919

# Test Prediction
y_pred = model3.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)

15/15 [==============================] - 0s 7ms/step

# Test Accuracy
accuracyScore3 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore3)

0.7726315789473684

Classification Report

# Compute the classification report
cr3 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

print('Classification report:\n',cr3)

Classification report:
                            precision    recall  f1-score   support

Small-flowered Cranesbill       0.83      0.88      0.85        50
                  Fat Hen       0.83      0.81      0.82        48
          Shepherds Purse       0.64      0.61      0.62        23
             Common wheat       0.83      0.45      0.59        22
         Common Chickweed       0.86      0.97      0.91        61
                 Charlock       0.86      0.92      0.89        39
                 Cleavers       0.91      0.72      0.81        29
        Scentless Mayweed       0.75      0.79      0.77        52
               Sugar beet       0.72      0.74      0.73        38
                    Maize       0.65      0.68      0.67        22
              Black-grass       0.35      0.23      0.28        26
         Loose Silky-bent       0.74      0.83      0.78        65

                 accuracy                           0.77       475
                macro avg       0.75      0.72      0.73       475
             weighted avg       0.77      0.77      0.76       475

Model3s overall accuracy is at 0.76, and the macro average F1-score is at 0.73, while the weighted average is at 0.7.

The model performs well on some classes, such as Small-flowered Cranesbill, Fat Hen, Common Chickweed, and Charlock, but doesn't perform well with others like Black-grass and Shepherds Purse.

Scoresdf = Scoresdf.append({'model':'Model3','accuracy':accuracyScore3,'layers':'10','total parameters':'3,306,188', 'neurons':'128', 'optimizer':'SGD'},ignore_index=True)

Confusion Matrix

# Computer the Confusion matrix
cm3 = confusion_matrix(normal_y_test, y_test_pred_classes)

plt.figure(figsize=(8,6))
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
sns.heatmap(cm3, xticklabels=name_classes, yticklabels=name_classes, annot=True)

<Axes: >

Overall, model2 performs better than model3.

Model 4

Lets try adding more layers to try to improve the previous models.

Layers : 17

Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu

Layer 2: BatchNormalization()

Layer 3: MaxPooling2D layer

Layer 4: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu

Layer 5: BatchNormalization()

Layer 6: MaxPooling2D layer

Layer 7: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu

Layer 8: BatchNormalization()

Layer 9: MaxPooling2D layer

Layer 7: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu

Layer 8: BatchNormalization()

Layer 9: MaxPooling2D layer

Layer 10: Flatten layer

Layer 11: Dense layer : 256 neurons, Relu

Layer 12: BatchNormalization

Layer 13: Droput layer: 0.5

Optimizer: Adam

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

Model Implementation

# initialized a sequential model
model4 = Sequential()

model4.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model4.add(BatchNormalization())
model4.add(MaxPooling2D(pool_size=(2,2)))

model4.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
model4.add(BatchNormalization())
model4.add(MaxPooling2D(pool_size=(2,2)))

model4.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model4.add(BatchNormalization())
model4.add(MaxPooling2D(pool_size=(2,2)))

model4.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model4.add(BatchNormalization())
model4.add(MaxPooling2D(pool_size=(2,2)))

model4.add(Flatten())
model4.add(Dense(128, activation='relu'))
model4.add(BatchNormalization())

model4.add(Dropout(0.5))
model4.add(Dense(num_classes, activation='softmax'))

# Compile the model
model4.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# Print the model summary
model4.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 126, 126, 32)      896       
                                                                 
 batch_normalization (BatchN  (None, 126, 126, 32)     128       
 ormalization)                                                   
                                                                 
 max_pooling2d (MaxPooling2D  (None, 63, 63, 32)       0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 61, 61, 64)        18496     
                                                                 
 batch_normalization_1 (Batc  (None, 61, 61, 64)       256       
 hNormalization)                                                 
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 30, 30, 64)       0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 28, 28, 128)       73856     
                                                                 
 batch_normalization_2 (Batc  (None, 28, 28, 128)      512       
 hNormalization)                                                 
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 14, 14, 128)      0         
 2D)                                                             
                                                                 
 conv2d_3 (Conv2D)           (None, 12, 12, 128)       147584    
                                                                 
 batch_normalization_3 (Batc  (None, 12, 12, 128)      512       
 hNormalization)                                                 
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 6, 6, 128)        0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 4608)              0         
                                                                 
 dense (Dense)               (None, 128)               589952    
                                                                 
 batch_normalization_4 (Batc  (None, 128)              512       
 hNormalization)                                                 
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_1 (Dense)             (None, 12)                1548      
                                                                 
=================================================================
Total params: 834,252
Trainable params: 833,292
Non-trainable params: 960
_________________________________________________________________

Model Visualization

#visualize model4 architecture
plot_model(model4, to_file='model4_plot.png', show_shapes=True, show_layer_names=True)

#visualize model4 architecture using visualkeras
visualkeras.layered_view(model4, legend=True)

Early Stopping

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history4=model4.fit(X_train,
          y_train_onehot,
          epochs=100,
          batch_size=32,validation_split=0.10,callbacks=[es, mc])

Epoch 1/100
121/121 [==============================] - ETA: 0s - loss: 1.6201 - accuracy: 0.5134
Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5
121/121 [==============================] - 9s 31ms/step - loss: 1.6201 - accuracy: 0.5134 - val_loss: 5.6225 - val_accuracy: 0.1215
Epoch 2/100
120/121 [============================>.] - ETA: 0s - loss: 0.8899 - accuracy: 0.7138
Epoch 2: val_accuracy did not improve from 0.12150
121/121 [==============================] - 3s 27ms/step - loss: 0.8915 - accuracy: 0.7135 - val_loss: 10.7526 - val_accuracy: 0.1215
Epoch 3/100
119/121 [============================>.] - ETA: 0s - loss: 0.6252 - accuracy: 0.7915
Epoch 3: val_accuracy improved from 0.12150 to 0.12383, saving model to best_model.h5
121/121 [==============================] - 3s 28ms/step - loss: 0.6285 - accuracy: 0.7902 - val_loss: 10.3552 - val_accuracy: 0.1238
Epoch 4/100
119/121 [============================>.] - ETA: 0s - loss: 0.5196 - accuracy: 0.8267
Epoch 4: val_accuracy improved from 0.12383 to 0.24299, saving model to best_model.h5
121/121 [==============================] - 4s 32ms/step - loss: 0.5195 - accuracy: 0.8269 - val_loss: 4.9083 - val_accuracy: 0.2430
Epoch 5/100
120/121 [============================>.] - ETA: 0s - loss: 0.4166 - accuracy: 0.8544
Epoch 5: val_accuracy improved from 0.24299 to 0.30374, saving model to best_model.h5
121/121 [==============================] - 3s 28ms/step - loss: 0.4166 - accuracy: 0.8547 - val_loss: 3.4493 - val_accuracy: 0.3037
Epoch 6/100
120/121 [============================>.] - ETA: 0s - loss: 0.3413 - accuracy: 0.8857
Epoch 6: val_accuracy improved from 0.30374 to 0.59579, saving model to best_model.h5
121/121 [==============================] - 3s 28ms/step - loss: 0.3429 - accuracy: 0.8854 - val_loss: 1.3685 - val_accuracy: 0.5958
Epoch 7/100
119/121 [============================>.] - ETA: 0s - loss: 0.2859 - accuracy: 0.9097
Epoch 7: val_accuracy improved from 0.59579 to 0.86682, saving model to best_model.h5
121/121 [==============================] - 3s 27ms/step - loss: 0.2853 - accuracy: 0.9098 - val_loss: 0.3992 - val_accuracy: 0.8668
Epoch 8/100
119/121 [============================>.] - ETA: 0s - loss: 0.2253 - accuracy: 0.9315
Epoch 8: val_accuracy did not improve from 0.86682
121/121 [==============================] - 4s 30ms/step - loss: 0.2259 - accuracy: 0.9306 - val_loss: 1.2766 - val_accuracy: 0.6332
Epoch 9/100
119/121 [============================>.] - ETA: 0s - loss: 0.1847 - accuracy: 0.9459
Epoch 9: val_accuracy did not improve from 0.86682
121/121 [==============================] - 3s 28ms/step - loss: 0.1850 - accuracy: 0.9459 - val_loss: 0.4840 - val_accuracy: 0.8248
Epoch 10/100
120/121 [============================>.] - ETA: 0s - loss: 0.1638 - accuracy: 0.9474
Epoch 10: val_accuracy did not improve from 0.86682
121/121 [==============================] - 3s 27ms/step - loss: 0.1647 - accuracy: 0.9470 - val_loss: 1.0341 - val_accuracy: 0.7009
Epoch 11/100
119/121 [============================>.] - ETA: 0s - loss: 0.1588 - accuracy: 0.9451
Epoch 11: val_accuracy did not improve from 0.86682
121/121 [==============================] - 3s 27ms/step - loss: 0.1594 - accuracy: 0.9449 - val_loss: 1.2241 - val_accuracy: 0.6799
Epoch 12/100
120/121 [============================>.] - ETA: 0s - loss: 0.1690 - accuracy: 0.9451
Epoch 12: val_accuracy did not improve from 0.86682
121/121 [==============================] - 3s 28ms/step - loss: 0.1699 - accuracy: 0.9444 - val_loss: 1.9335 - val_accuracy: 0.5818
Epoch 13/100
121/121 [==============================] - ETA: 0s - loss: 0.1456 - accuracy: 0.9485
Epoch 13: val_accuracy did not improve from 0.86682
121/121 [==============================] - 3s 29ms/step - loss: 0.1456 - accuracy: 0.9485 - val_loss: 0.7255 - val_accuracy: 0.7687
Epoch 14/100
120/121 [============================>.] - ETA: 0s - loss: 0.1349 - accuracy: 0.9568
Epoch 14: val_accuracy improved from 0.86682 to 0.86916, saving model to best_model.h5
121/121 [==============================] - 3s 27ms/step - loss: 0.1356 - accuracy: 0.9563 - val_loss: 0.4801 - val_accuracy: 0.8692
Epoch 15/100
120/121 [============================>.] - ETA: 0s - loss: 0.1090 - accuracy: 0.9648
Epoch 15: val_accuracy did not improve from 0.86916
121/121 [==============================] - 3s 27ms/step - loss: 0.1103 - accuracy: 0.9646 - val_loss: 0.5455 - val_accuracy: 0.8178
Epoch 16/100
120/121 [============================>.] - ETA: 0s - loss: 0.0723 - accuracy: 0.9812
Epoch 16: val_accuracy did not improve from 0.86916
121/121 [==============================] - 3s 27ms/step - loss: 0.0723 - accuracy: 0.9813 - val_loss: 0.6476 - val_accuracy: 0.7944
Epoch 17/100
119/121 [============================>.] - ETA: 0s - loss: 0.0633 - accuracy: 0.9819
Epoch 17: val_accuracy did not improve from 0.86916
121/121 [==============================] - 3s 29ms/step - loss: 0.0629 - accuracy: 0.9821 - val_loss: 1.0993 - val_accuracy: 0.7243
Epoch 17: early stopping

Training stopped on epoch 33. The highest validation accuracy was 0.8925 where the validation loss was 0.3573.

Model4 Accuracy vs Epoch curve

plt.plot(history4.history['accuracy'])
plt.plot(history4.history['val_accuracy'])
plt.title('model4 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

Notice that the model tends to overfit as epochs increased.

Evaluation of model4 model.evaluate using the Test data

# Evaluate the model on the test data
score4 = model4.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score4[0])
print('Test accuracy:', score4[1])

Test loss: 1.3278809785842896
Test accuracy: 0.7052631378173828

# Test Prediction
y_pred = model4.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)

15/15 [==============================] - 0s 8ms/step

# Test Accuracy
accuracyScore4 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore4)

0.7052631578947368

Classification Report

# Compute the classification report
cr4 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

print('Classification report:\n',cr4)

Classification report:
                            precision    recall  f1-score   support

Small-flowered Cranesbill       0.95      0.78      0.86        50
                  Fat Hen       0.47      0.98      0.64        48
          Shepherds Purse       0.53      0.43      0.48        23
             Common wheat       0.82      0.41      0.55        22
         Common Chickweed       0.91      0.82      0.86        61
                 Charlock       1.00      0.82      0.90        39
                 Cleavers       0.96      0.86      0.91        29
        Scentless Mayweed       1.00      0.33      0.49        52
               Sugar beet       0.85      0.76      0.81        38
                    Maize       0.51      0.95      0.67        22
              Black-grass       0.25      0.27      0.26        26
         Loose Silky-bent       0.68      0.75      0.72        65

                 accuracy                           0.71       475
                macro avg       0.74      0.68      0.68       475
             weighted avg       0.78      0.71      0.70       475

Model4 has an overall accuracy of 83%.

The precision scores range from 0.48 to 0.95, indicating that the model's ability to correctly identify positive instances varies by class. For example, the model has high precision in identifying Charlock and Cleavers, while it has relatively lower precision in identifying Black-grass and Shepherds Purse.

The recall scores range from 0.50 to 0.97, indicating that the model's ability to correctly identify positive instances also varies by class. For example, the model has high recall in identifying Common Chickweed and Charlock, while it has relatively lower recall in identifying Black-grass and Shepherds Purse.

The F1-scores are a harmonic mean of precision and recall and range from 0.49 to 0.92. The F1-score provides a balance between precision and recall, indicating the overall performance of the model for each class.

Scoresdf = Scoresdf.append({'model':'Model4','accuracy':accuracyScore4,'layers':'17','total parameters':'834,252', 'neurons':'128', 'optimizer':'Adam'},ignore_index=True)

Confusion Matrix

# Compute Confusion matrix
cm4 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm4, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()

Model4 seems to perform well overall, but there is some variation in the performance of the model for different classes.

Model 5

To continue trying to improve the previous model, we add more layers to our model.

Layers : 19

Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu

Layer 2: BatchNormalization()

Layer 3: MaxPooling2D layer

Layer 4: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu

Layer 5: BatchNormalization()

Layer 6: MaxPooling2D layer

Layer 7: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu

Layer 8: BatchNormalization()

Layer 9: MaxPooling2D layer

Layer 10: Convolutional layer : 256 filters, Kernel_size = (3,3), Relu

Layer 11: BatchNormalization()

Layer 12: MaxPooling2D layer

Layer 13: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu

Layer 14: BatchNormalization()

Layer 15: MaxPooling2D layer

Layer 16: Flatten layer

Layer 17: Dense layer : 256 neurons, Relu

Layer 18: BatchNormalization

Layer 19: Droput layer: 0.5

Optimizer: Adam

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

Model Implementation

# initialized a sequential model
model5 = Sequential()

model5.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))

model5.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))

model5.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))

model5.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))

model5.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model5.add(BatchNormalization())
model5.add(MaxPooling2D(pool_size=(2,2)))

model5.add(Flatten())
model5.add(Dense(256, activation='relu'))
model5.add(BatchNormalization())
model5.add(Dropout(0.5))

model5.add(Dense(num_classes, activation='softmax'))

# Compile the model
model5.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# Print the model summary
model5.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 126, 126, 32)      896       
                                                                 
 batch_normalization (BatchN  (None, 126, 126, 32)     128       
 ormalization)                                                   
                                                                 
 max_pooling2d (MaxPooling2D  (None, 63, 63, 32)       0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 61, 61, 64)        18496     
                                                                 
 batch_normalization_1 (Batc  (None, 61, 61, 64)       256       
 hNormalization)                                                 
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 30, 30, 64)       0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 28, 28, 128)       73856     
                                                                 
 batch_normalization_2 (Batc  (None, 28, 28, 128)      512       
 hNormalization)                                                 
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 14, 14, 128)      0         
 2D)                                                             
                                                                 
 conv2d_3 (Conv2D)           (None, 12, 12, 256)       295168    
                                                                 
 batch_normalization_3 (Batc  (None, 12, 12, 256)      1024      
 hNormalization)                                                 
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 6, 6, 256)        0         
 2D)                                                             
                                                                 
 conv2d_4 (Conv2D)           (None, 4, 4, 128)         295040    
                                                                 
 batch_normalization_4 (Batc  (None, 4, 4, 128)        512       
 hNormalization)                                                 
                                                                 
 max_pooling2d_4 (MaxPooling  (None, 2, 2, 128)        0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 512)               0         
                                                                 
 dense (Dense)               (None, 256)               131328    
                                                                 
 batch_normalization_5 (Batc  (None, 256)              1024      
 hNormalization)                                                 
                                                                 
 dropout (Dropout)           (None, 256)               0         
                                                                 
 dense_1 (Dense)             (None, 12)                3084      
                                                                 
=================================================================
Total params: 821,324
Trainable params: 819,596
Non-trainable params: 1,728
_________________________________________________________________

Model5 Architecture Visualization

##visualize model architecture
plot_model(model5, to_file='model5_plot.png', show_shapes=True, show_layer_names=True)

##model architecture visualization using visualkeras
visualkeras.layered_view(model5, legend=True)

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=5)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history5 = model5.fit(X_train,
          y_train_onehot,
          epochs=100,
          batch_size=32,validation_split=0.10,callbacks=[es, mc])

Epoch 1/100
119/121 [============================>.] - ETA: 0s - loss: 1.6716 - accuracy: 0.5011
Epoch 1: val_accuracy improved from -inf to 0.06308, saving model to best_model.h5
121/121 [==============================] - 9s 36ms/step - loss: 1.6664 - accuracy: 0.5032 - val_loss: 7.5329 - val_accuracy: 0.0631
Epoch 2/100
119/121 [============================>.] - ETA: 0s - loss: 0.8780 - accuracy: 0.7124
Epoch 2: val_accuracy improved from 0.06308 to 0.12150, saving model to best_model.h5
121/121 [==============================] - 4s 32ms/step - loss: 0.8768 - accuracy: 0.7130 - val_loss: 13.4861 - val_accuracy: 0.1215
Epoch 3/100
119/121 [============================>.] - ETA: 0s - loss: 0.6572 - accuracy: 0.7723
Epoch 3: val_accuracy improved from 0.12150 to 0.14486, saving model to best_model.h5
121/121 [==============================] - 4s 30ms/step - loss: 0.6564 - accuracy: 0.7723 - val_loss: 12.1352 - val_accuracy: 0.1449
Epoch 4/100
120/121 [============================>.] - ETA: 0s - loss: 0.4920 - accuracy: 0.8268
Epoch 4: val_accuracy improved from 0.14486 to 0.15888, saving model to best_model.h5
121/121 [==============================] - 4s 29ms/step - loss: 0.4933 - accuracy: 0.8264 - val_loss: 7.2529 - val_accuracy: 0.1589
Epoch 5/100
119/121 [============================>.] - ETA: 0s - loss: 0.4023 - accuracy: 0.8595
Epoch 5: val_accuracy improved from 0.15888 to 0.54907, saving model to best_model.h5
121/121 [==============================] - 4s 32ms/step - loss: 0.4035 - accuracy: 0.8594 - val_loss: 1.6262 - val_accuracy: 0.5491
Epoch 6/100
120/121 [============================>.] - ETA: 0s - loss: 0.3601 - accuracy: 0.8763
Epoch 6: val_accuracy improved from 0.54907 to 0.72664, saving model to best_model.h5
121/121 [==============================] - 4s 31ms/step - loss: 0.3619 - accuracy: 0.8760 - val_loss: 0.8660 - val_accuracy: 0.7266
Epoch 7/100
120/121 [============================>.] - ETA: 0s - loss: 0.2981 - accuracy: 0.8898
Epoch 7: val_accuracy did not improve from 0.72664
121/121 [==============================] - 3s 28ms/step - loss: 0.2979 - accuracy: 0.8900 - val_loss: 0.9416 - val_accuracy: 0.6963
Epoch 8/100
119/121 [============================>.] - ETA: 0s - loss: 0.2452 - accuracy: 0.9136
Epoch 8: val_accuracy improved from 0.72664 to 0.78271, saving model to best_model.h5
121/121 [==============================] - 3s 29ms/step - loss: 0.2483 - accuracy: 0.9127 - val_loss: 0.6978 - val_accuracy: 0.7827
Epoch 9/100
119/121 [============================>.] - ETA: 0s - loss: 0.2397 - accuracy: 0.9107
Epoch 9: val_accuracy did not improve from 0.78271
121/121 [==============================] - 4s 30ms/step - loss: 0.2394 - accuracy: 0.9108 - val_loss: 1.1052 - val_accuracy: 0.6776
Epoch 10/100
119/121 [============================>.] - ETA: 0s - loss: 0.1977 - accuracy: 0.9244
Epoch 10: val_accuracy improved from 0.78271 to 0.81542, saving model to best_model.h5
121/121 [==============================] - 4s 33ms/step - loss: 0.1987 - accuracy: 0.9236 - val_loss: 0.5499 - val_accuracy: 0.8154
Epoch 11/100
119/121 [============================>.] - ETA: 0s - loss: 0.1850 - accuracy: 0.9288
Epoch 11: val_accuracy improved from 0.81542 to 0.83879, saving model to best_model.h5
121/121 [==============================] - 4s 29ms/step - loss: 0.1846 - accuracy: 0.9288 - val_loss: 0.5094 - val_accuracy: 0.8388
Epoch 12/100
119/121 [============================>.] - ETA: 0s - loss: 0.1423 - accuracy: 0.9533
Epoch 12: val_accuracy improved from 0.83879 to 0.84112, saving model to best_model.h5
121/121 [==============================] - 3s 29ms/step - loss: 0.1429 - accuracy: 0.9530 - val_loss: 0.5343 - val_accuracy: 0.8411
Epoch 13/100
120/121 [============================>.] - ETA: 0s - loss: 0.1648 - accuracy: 0.9406
Epoch 13: val_accuracy did not improve from 0.84112
121/121 [==============================] - 4s 30ms/step - loss: 0.1648 - accuracy: 0.9407 - val_loss: 0.7414 - val_accuracy: 0.7804
Epoch 14/100
119/121 [============================>.] - ETA: 0s - loss: 0.1324 - accuracy: 0.9554
Epoch 14: val_accuracy did not improve from 0.84112
121/121 [==============================] - 4s 31ms/step - loss: 0.1320 - accuracy: 0.9558 - val_loss: 0.7250 - val_accuracy: 0.8107
Epoch 15/100
119/121 [============================>.] - ETA: 0s - loss: 0.1177 - accuracy: 0.9624
Epoch 15: val_accuracy did not improve from 0.84112
121/121 [==============================] - 3s 28ms/step - loss: 0.1198 - accuracy: 0.9618 - val_loss: 0.7258 - val_accuracy: 0.7944
Epoch 16/100
120/121 [============================>.] - ETA: 0s - loss: 0.1284 - accuracy: 0.9544
Epoch 16: val_accuracy did not improve from 0.84112
121/121 [==============================] - 3s 28ms/step - loss: 0.1285 - accuracy: 0.9543 - val_loss: 0.4891 - val_accuracy: 0.8411
Epoch 17/100
120/121 [============================>.] - ETA: 0s - loss: 0.1123 - accuracy: 0.9620
Epoch 17: val_accuracy did not improve from 0.84112
121/121 [==============================] - 3s 28ms/step - loss: 0.1122 - accuracy: 0.9620 - val_loss: 0.6232 - val_accuracy: 0.8014
Epoch 18/100
119/121 [============================>.] - ETA: 0s - loss: 0.0725 - accuracy: 0.9764
Epoch 18: val_accuracy improved from 0.84112 to 0.85748, saving model to best_model.h5
121/121 [==============================] - 4s 32ms/step - loss: 0.0727 - accuracy: 0.9763 - val_loss: 0.4822 - val_accuracy: 0.8575
Epoch 19/100
119/121 [============================>.] - ETA: 0s - loss: 0.0758 - accuracy: 0.9751
Epoch 19: val_accuracy did not improve from 0.85748
121/121 [==============================] - 4s 30ms/step - loss: 0.0755 - accuracy: 0.9750 - val_loss: 0.4922 - val_accuracy: 0.8458
Epoch 20/100
120/121 [============================>.] - ETA: 0s - loss: 0.0633 - accuracy: 0.9794
Epoch 20: val_accuracy did not improve from 0.85748
121/121 [==============================] - 3s 28ms/step - loss: 0.0646 - accuracy: 0.9789 - val_loss: 1.2289 - val_accuracy: 0.6822
Epoch 21/100
120/121 [============================>.] - ETA: 0s - loss: 0.1150 - accuracy: 0.9578
Epoch 21: val_accuracy did not improve from 0.85748
121/121 [==============================] - 3s 28ms/step - loss: 0.1156 - accuracy: 0.9576 - val_loss: 0.6274 - val_accuracy: 0.8201
Epoch 22/100
119/121 [============================>.] - ETA: 0s - loss: 0.0844 - accuracy: 0.9688
Epoch 22: val_accuracy did not improve from 0.85748
121/121 [==============================] - 4s 30ms/step - loss: 0.0837 - accuracy: 0.9691 - val_loss: 1.7372 - val_accuracy: 0.6986
Epoch 23/100
119/121 [============================>.] - ETA: 0s - loss: 0.0455 - accuracy: 0.9863
Epoch 23: val_accuracy improved from 0.85748 to 0.87617, saving model to best_model.h5
121/121 [==============================] - 4s 30ms/step - loss: 0.0454 - accuracy: 0.9865 - val_loss: 0.4484 - val_accuracy: 0.8762
Epoch 24/100
120/121 [============================>.] - ETA: 0s - loss: 0.0505 - accuracy: 0.9854
Epoch 24: val_accuracy improved from 0.87617 to 0.87850, saving model to best_model.h5
121/121 [==============================] - 4s 29ms/step - loss: 0.0504 - accuracy: 0.9854 - val_loss: 0.3818 - val_accuracy: 0.8785
Epoch 25/100
120/121 [============================>.] - ETA: 0s - loss: 0.0419 - accuracy: 0.9878
Epoch 25: val_accuracy did not improve from 0.87850
121/121 [==============================] - 3s 28ms/step - loss: 0.0423 - accuracy: 0.9875 - val_loss: 0.7614 - val_accuracy: 0.8248
Epoch 26/100
119/121 [============================>.] - ETA: 0s - loss: 0.0427 - accuracy: 0.9856
Epoch 26: val_accuracy did not improve from 0.87850
121/121 [==============================] - 4s 31ms/step - loss: 0.0425 - accuracy: 0.9854 - val_loss: 0.8043 - val_accuracy: 0.8271
Epoch 27/100
120/121 [============================>.] - ETA: 0s - loss: 0.0470 - accuracy: 0.9844
Epoch 27: val_accuracy did not improve from 0.87850
121/121 [==============================] - 4s 31ms/step - loss: 0.0473 - accuracy: 0.9841 - val_loss: 3.3697 - val_accuracy: 0.5093
Epoch 28/100
120/121 [============================>.] - ETA: 0s - loss: 0.0518 - accuracy: 0.9844
Epoch 28: val_accuracy did not improve from 0.87850
121/121 [==============================] - 3s 28ms/step - loss: 0.0518 - accuracy: 0.9844 - val_loss: 0.5574 - val_accuracy: 0.8435
Epoch 29/100
120/121 [============================>.] - ETA: 0s - loss: 0.0408 - accuracy: 0.9875
Epoch 29: val_accuracy did not improve from 0.87850
121/121 [==============================] - 3s 28ms/step - loss: 0.0407 - accuracy: 0.9875 - val_loss: 1.0695 - val_accuracy: 0.7734
Epoch 29: early stopping

Training stopped at epoch 15. The highest validation accuracy was 0.8575 and validation loss is 0.4383.

Accuracy vs Epoch curve

plt.plot(history5.history['accuracy'])
plt.plot(history5.history['val_accuracy'])
plt.title('model5 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

The model is overfitting. However, around epoch 9, it has a high validation accuracy and a high training accuracy.

Evaluation of model5 model.evaluate using the Test data

# Evaluate the model on the test data
score5 = model5.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score5[0])
print('Test accuracy:', score5[1])

Test loss: 1.1474685668945312
Test accuracy: 0.7557894587516785

# Test Prediction
y_pred = model5.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)

15/15 [==============================] - 0s 11ms/step

# Test Accuracy
accuracyScore5 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore5)

0.7557894736842106

Classification Report

# Compute the classification report
cr5 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

print('Classification report:\n',cr5)

Classification report:
                            precision    recall  f1-score   support

Small-flowered Cranesbill       0.94      0.92      0.93        50
                  Fat Hen       0.95      0.85      0.90        48
          Shepherds Purse       0.81      0.74      0.77        23
             Common wheat       0.82      0.64      0.72        22
         Common Chickweed       0.98      0.82      0.89        61
                 Charlock       0.97      0.90      0.93        39
                 Cleavers       0.93      0.86      0.89        29
        Scentless Mayweed       0.44      0.92      0.60        52
               Sugar beet       1.00      0.21      0.35        38
                    Maize       0.53      0.77      0.63        22
              Black-grass       0.64      0.27      0.38        26
         Loose Silky-bent       0.71      0.78      0.74        65

                 accuracy                           0.76       475
                macro avg       0.81      0.72      0.73       475
             weighted avg       0.82      0.76      0.75       475

The accuracy of the model5 is 83%, and the weighted average F1-score is 0.82. The model performs relatively well in predicting most of the plant species, with high precision, recall and f1-scores, except for Black-grass.

However Model4 seems to be performing slightly better.

Scoresdf = Scoresdf.append({'model':'Model5','accuracy':accuracyScore5,'layers':'20','total parameters':'821,324', 'neurons':'256', 'optimizer':'Adam'},ignore_index=True)

Confusion Matrix

#Visualize the Confusion matrix
cm5 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))

sns.heatmap(cm5, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()

Model5 performs well for most classes but Black-grass.

Model 6

To try to improve the performance of our previous models, we changethe layers and filters.

Layers: 16

• Layer 1: Convolution, filter: 64, kernel 3,3
• Layer 2: BatchNormalization
• Layer 3: MaxPooling2D
• Layer 4: Convolution, filter 128, kernel 3,3
• Layer 5: BatchNormalization
• Layer 6: MaxPooling2D
• Layer 7: Convolution, filter 256, kernel 3,3
• Layer 8: BatchNormalization
• Layer 9: MaxPooling2D
• Layer 10: Convolution, filter 512, kernel 3,3
• Layer 11: BatchNormalization
• Layer 12: MaxPooling2D
• Layer 13: Flatten
• Layer 14: Dense 256
• Layer 15: BatchNormalization
• Layer 16: Dropout 0.5

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

Model Implementation

# initialized a sequential model
model6 = Sequential()

model6.add(Conv2D(64, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model6.add(BatchNormalization())
model6.add(MaxPooling2D(pool_size=(2,2)))

model6.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model6.add(BatchNormalization())
model6.add(MaxPooling2D(pool_size=(2,2)))

model6.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
model6.add(BatchNormalization())
model6.add(MaxPooling2D(pool_size=(2,2)))

model6.add(Flatten())
model6.add(Dense(512, activation='relu'))
model6.add(BatchNormalization())
model6.add(Dropout(0.5))

model6.add(Dense(256, activation='relu'))
model6.add(BatchNormalization())
model6.add(Dropout(0.5))

model6.add(Dense(num_classes, activation='softmax'))

# Compile the model
model6.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# Print the model summary
model6.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 126, 126, 64)      1792      
                                                                 
 batch_normalization (BatchN  (None, 126, 126, 64)     256       
 ormalization)                                                   
                                                                 
 max_pooling2d (MaxPooling2D  (None, 63, 63, 64)       0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 61, 61, 128)       73856     
                                                                 
 batch_normalization_1 (Batc  (None, 61, 61, 128)      512       
 hNormalization)                                                 
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 30, 30, 128)      0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 28, 28, 256)       295168    
                                                                 
 batch_normalization_2 (Batc  (None, 28, 28, 256)      1024      
 hNormalization)                                                 
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 14, 14, 256)      0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 50176)             0         
                                                                 
 dense (Dense)               (None, 512)               25690624  
                                                                 
 batch_normalization_3 (Batc  (None, 512)              2048      
 hNormalization)                                                 
                                                                 
 dropout (Dropout)           (None, 512)               0         
                                                                 
 dense_1 (Dense)             (None, 256)               131328    
                                                                 
 batch_normalization_4 (Batc  (None, 256)              1024      
 hNormalization)                                                 
                                                                 
 dropout_1 (Dropout)         (None, 256)               0         
                                                                 
 dense_2 (Dense)             (None, 12)                3084      
                                                                 
=================================================================
Total params: 26,200,716
Trainable params: 26,198,284
Non-trainable params: 2,432
_________________________________________________________________

Model6 Architecture Visualization

#visualize model6 architecture
plot_model(model6, to_file='model6_plot.png', show_shapes=True, show_layer_names=True)

#visualize model6 architecture using visualkeras
visualkeras.layered_view(model6, legend=True)

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 40 epochs and validation_split as 10%
history6=model6.fit(X_train,
          y_train_onehot,
          epochs=40,
          batch_size=32,validation_split=0.10,callbacks=[es, mc])

Epoch 1/40
120/121 [============================>.] - ETA: 0s - loss: 1.9669 - accuracy: 0.4292
Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5
121/121 [==============================] - 18s 98ms/step - loss: 1.9666 - accuracy: 0.4292 - val_loss: 7.0356 - val_accuracy: 0.1215
Epoch 2/40
120/121 [============================>.] - ETA: 0s - loss: 1.1810 - accuracy: 0.6206
Epoch 2: val_accuracy did not improve from 0.12150
121/121 [==============================] - 6s 53ms/step - loss: 1.1832 - accuracy: 0.6200 - val_loss: 8.4802 - val_accuracy: 0.1215
Epoch 3/40
120/121 [============================>.] - ETA: 0s - loss: 0.8575 - accuracy: 0.7164
Epoch 3: val_accuracy improved from 0.12150 to 0.12383, saving model to best_model.h5
121/121 [==============================] - 12s 99ms/step - loss: 0.8580 - accuracy: 0.7161 - val_loss: 6.8681 - val_accuracy: 0.1238
Epoch 4/40
121/121 [==============================] - ETA: 0s - loss: 0.6646 - accuracy: 0.7720
Epoch 4: val_accuracy improved from 0.12383 to 0.20794, saving model to best_model.h5
121/121 [==============================] - 13s 109ms/step - loss: 0.6646 - accuracy: 0.7720 - val_loss: 5.8048 - val_accuracy: 0.2079
Epoch 5/40
121/121 [==============================] - ETA: 0s - loss: 0.5248 - accuracy: 0.8131
Epoch 5: val_accuracy improved from 0.20794 to 0.42290, saving model to best_model.h5
121/121 [==============================] - 12s 102ms/step - loss: 0.5248 - accuracy: 0.8131 - val_loss: 2.2491 - val_accuracy: 0.4229
Epoch 6/40
121/121 [==============================] - ETA: 0s - loss: 0.4506 - accuracy: 0.8451
Epoch 6: val_accuracy improved from 0.42290 to 0.58879, saving model to best_model.h5
121/121 [==============================] - 12s 100ms/step - loss: 0.4506 - accuracy: 0.8451 - val_loss: 1.3419 - val_accuracy: 0.5888
Epoch 7/40
121/121 [==============================] - ETA: 0s - loss: 0.3584 - accuracy: 0.8713
Epoch 7: val_accuracy improved from 0.58879 to 0.62150, saving model to best_model.h5
121/121 [==============================] - 12s 101ms/step - loss: 0.3584 - accuracy: 0.8713 - val_loss: 1.1451 - val_accuracy: 0.6215
Epoch 8/40
120/121 [============================>.] - ETA: 0s - loss: 0.3424 - accuracy: 0.8823
Epoch 8: val_accuracy improved from 0.62150 to 0.76636, saving model to best_model.h5
121/121 [==============================] - 11s 94ms/step - loss: 0.3438 - accuracy: 0.8815 - val_loss: 0.7611 - val_accuracy: 0.7664
Epoch 9/40
121/121 [==============================] - ETA: 0s - loss: 0.3091 - accuracy: 0.8895
Epoch 9: val_accuracy did not improve from 0.76636
121/121 [==============================] - 6s 52ms/step - loss: 0.3091 - accuracy: 0.8895 - val_loss: 2.2853 - val_accuracy: 0.3715
Epoch 10/40
120/121 [============================>.] - ETA: 0s - loss: 0.3499 - accuracy: 0.8802
Epoch 10: val_accuracy did not improve from 0.76636
121/121 [==============================] - 6s 52ms/step - loss: 0.3516 - accuracy: 0.8799 - val_loss: 6.1112 - val_accuracy: 0.1706
Epoch 11/40
120/121 [============================>.] - ETA: 0s - loss: 0.2723 - accuracy: 0.9057
Epoch 11: val_accuracy did not improve from 0.76636
121/121 [==============================] - 6s 52ms/step - loss: 0.2727 - accuracy: 0.9054 - val_loss: 0.9662 - val_accuracy: 0.7173
Epoch 12/40
120/121 [============================>.] - ETA: 0s - loss: 0.1859 - accuracy: 0.9315
Epoch 12: val_accuracy improved from 0.76636 to 0.82944, saving model to best_model.h5
121/121 [==============================] - 9s 71ms/step - loss: 0.1868 - accuracy: 0.9314 - val_loss: 0.5295 - val_accuracy: 0.8294
Epoch 13/40
120/121 [============================>.] - ETA: 0s - loss: 0.1668 - accuracy: 0.9398
Epoch 13: val_accuracy did not improve from 0.82944
121/121 [==============================] - 6s 53ms/step - loss: 0.1668 - accuracy: 0.9400 - val_loss: 0.8630 - val_accuracy: 0.7617
Epoch 14/40
120/121 [============================>.] - ETA: 0s - loss: 0.1245 - accuracy: 0.9604
Epoch 14: val_accuracy did not improve from 0.82944
121/121 [==============================] - 6s 51ms/step - loss: 0.1244 - accuracy: 0.9605 - val_loss: 7.8960 - val_accuracy: 0.2243
Epoch 15/40
120/121 [============================>.] - ETA: 0s - loss: 0.1046 - accuracy: 0.9672
Epoch 15: val_accuracy did not improve from 0.82944
121/121 [==============================] - 6s 53ms/step - loss: 0.1080 - accuracy: 0.9670 - val_loss: 0.7944 - val_accuracy: 0.7640
Epoch 16/40
120/121 [============================>.] - ETA: 0s - loss: 0.0991 - accuracy: 0.9680
Epoch 16: val_accuracy did not improve from 0.82944
121/121 [==============================] - 6s 51ms/step - loss: 0.0990 - accuracy: 0.9680 - val_loss: 0.7597 - val_accuracy: 0.7780
Epoch 17/40
120/121 [============================>.] - ETA: 0s - loss: 0.0788 - accuracy: 0.9742
Epoch 17: val_accuracy did not improve from 0.82944
121/121 [==============================] - 6s 53ms/step - loss: 0.0787 - accuracy: 0.9743 - val_loss: 0.6409 - val_accuracy: 0.8224
Epoch 18/40
121/121 [==============================] - ETA: 0s - loss: 0.0746 - accuracy: 0.9727
Epoch 18: val_accuracy did not improve from 0.82944
121/121 [==============================] - 6s 53ms/step - loss: 0.0746 - accuracy: 0.9727 - val_loss: 0.9586 - val_accuracy: 0.7383
Epoch 19/40
120/121 [============================>.] - ETA: 0s - loss: 0.0653 - accuracy: 0.9773
Epoch 19: val_accuracy did not improve from 0.82944
121/121 [==============================] - 6s 52ms/step - loss: 0.0652 - accuracy: 0.9774 - val_loss: 1.3687 - val_accuracy: 0.6706
Epoch 20/40
121/121 [==============================] - ETA: 0s - loss: 0.0584 - accuracy: 0.9784
Epoch 20: val_accuracy did not improve from 0.82944
121/121 [==============================] - 6s 53ms/step - loss: 0.0584 - accuracy: 0.9784 - val_loss: 1.8634 - val_accuracy: 0.6565
Epoch 21/40
120/121 [============================>.] - ETA: 0s - loss: 0.1058 - accuracy: 0.9643
Epoch 21: val_accuracy did not improve from 0.82944
121/121 [==============================] - 6s 50ms/step - loss: 0.1058 - accuracy: 0.9644 - val_loss: 1.1467 - val_accuracy: 0.7290
Epoch 22/40
120/121 [============================>.] - ETA: 0s - loss: 0.0926 - accuracy: 0.9688
Epoch 22: val_accuracy did not improve from 0.82944
121/121 [==============================] - 6s 53ms/step - loss: 0.0926 - accuracy: 0.9688 - val_loss: 4.7129 - val_accuracy: 0.4650
Epoch 22: early stopping

Training stopped at epoch 24. The highest Validation accuracy was 0.8481 and validation loss 0.5729.

Model6 Accuracy vs Epoch curve

plt.plot(history6.history['accuracy'])
plt.plot(history6.history['val_accuracy'])
plt.title('model6 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

This model seems to be overfitting.

Evaluation of model6 model.evaluate using the Test data

# Evaluate the model on the test data
score6 = model6.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score6[0])
print('Test accuracy:', score6[1])

Test loss: 4.940274238586426
Test accuracy: 0.42105263471603394

# Test Prediction
y_pred = model6.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)

15/15 [==============================] - 0s 13ms/step

# Test Accuracy
accuracyScore6 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore6)

0.42105263157894735

Classification Report

# Compute the classification report
cr6 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

print('Classification report:\n',cr6)

Classification report:
                            precision    recall  f1-score   support

Small-flowered Cranesbill       1.00      0.46      0.63        50
                  Fat Hen       1.00      0.08      0.15        48
          Shepherds Purse       0.57      0.17      0.27        23
             Common wheat       0.00      0.00      0.00        22
         Common Chickweed       0.94      0.56      0.70        61
                 Charlock       1.00      0.10      0.19        39
                 Cleavers       1.00      0.07      0.13        29
        Scentless Mayweed       0.22      1.00      0.36        52
               Sugar beet       1.00      0.03      0.05        38
                    Maize       0.33      0.64      0.43        22
              Black-grass       0.50      0.12      0.19        26
         Loose Silky-bent       0.54      0.91      0.67        65

                 accuracy                           0.42       475
                macro avg       0.67      0.34      0.31       475
             weighted avg       0.72      0.42      0.37       475

The model seems to be overfitting as it has high accuracy on the training set but low accuracy on the test set. The precision, recall, and F1-score for each class also vary widely, with some classes having high scores and others having low scores. This suggests that the model is not performing well in some classes.

Scoresdf = Scoresdf.append({'model':'Model6','accuracy':accuracyScore6,'layers':'17','total parameters':'26,200,716', 'neurons':'256', 'optimizer':'Adam'},ignore_index=True)

Confusion Matrix

# Calculate Confusion matrix
cm6 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm5, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()

Overall, Model6 doesn't perform well for some classes and is overfitting.

Model 7

To continue trying to improve our model, we added more layers and modified the filter sizes.

• Conv2D layer with 32 filters, a 3x3 kernel size, and ReLU activation.
• Conv2D layer with 32 filters, a 3x3 kernel size, and ReLU activation.
• BatchNormalization layer to normalize the output of the previous layers.
• MaxPooling2D layer with a 2x2 pool size to reduce the spatial dimensions of the output.
• Dropout layer to randomly set a fraction of the input units to 0 to prevent overfitting.
• Conv2D layer with 64 filters, a 3x3 kernel size, and ReLU activation.
• Conv2D layer with 64 filters, a 3x3 kernel size, and ReLU activation.
• BatchNormalization layer.
• MaxPooling2D layer.
• Dropout layer.
• Conv2D layer with 128 filters, a 3x3 kernel size, and ReLU activation.
• Conv2D layer with 128 filters, a 3x3 kernel size, and ReLU activation.
• BatchNormalization layer.
• MaxPooling2D layer.
• Dropout layer.
• Flatten layer to convert the output of the previous layers to a 1D vector.
• Dense layer with 512 units and ReLU activation.
• BatchNormalization layer.
• Dropout layer.

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

Model Implementation

#Model implementation
model7 = Sequential()

model7.add(Conv2D(32, (3, 3), activation='relu', input_shape=input_shape))
model7.add(Conv2D(32, (3, 3), activation='relu'))
model7.add(BatchNormalization())
model7.add(MaxPooling2D(pool_size=(2, 2)))
model7.add(Dropout(0.25))

model7.add(Conv2D(64, (3, 3), activation='relu'))
model7.add(Conv2D(64, (3, 3), activation='relu'))
model7.add(BatchNormalization())
model7.add(MaxPooling2D(pool_size=(2, 2)))
model7.add(Dropout(0.25))

model7.add(Conv2D(128, (3, 3), activation='relu'))
model7.add(Conv2D(128, (3, 3), activation='relu'))
model7.add(BatchNormalization())
model7.add(MaxPooling2D(pool_size=(2, 2)))
model7.add(Dropout(0.25))

model7.add(Flatten())
model7.add(Dense(512, activation='relu'))
model7.add(BatchNormalization())
model7.add(Dropout(0.5))
model7.add(Dense(num_classes, activation='softmax'))

model7.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

model7.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 126, 126, 32)      896       
                                                                 
 conv2d_1 (Conv2D)           (None, 124, 124, 32)      9248      
                                                                 
 batch_normalization (BatchN  (None, 124, 124, 32)     128       
 ormalization)                                                   
                                                                 
 max_pooling2d (MaxPooling2D  (None, 62, 62, 32)       0         
 )                                                               
                                                                 
 dropout (Dropout)           (None, 62, 62, 32)        0         
                                                                 
 conv2d_2 (Conv2D)           (None, 60, 60, 64)        18496     
                                                                 
 conv2d_3 (Conv2D)           (None, 58, 58, 64)        36928     
                                                                 
 batch_normalization_1 (Batc  (None, 58, 58, 64)       256       
 hNormalization)                                                 
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 29, 29, 64)       0         
 2D)                                                             
                                                                 
 dropout_1 (Dropout)         (None, 29, 29, 64)        0         
                                                                 
 conv2d_4 (Conv2D)           (None, 27, 27, 128)       73856     
                                                                 
 conv2d_5 (Conv2D)           (None, 25, 25, 128)       147584    
                                                                 
 batch_normalization_2 (Batc  (None, 25, 25, 128)      512       
 hNormalization)                                                 
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 12, 12, 128)      0         
 2D)                                                             
                                                                 
 dropout_2 (Dropout)         (None, 12, 12, 128)       0         
                                                                 
 flatten (Flatten)           (None, 18432)             0         
                                                                 
 dense (Dense)               (None, 512)               9437696   
                                                                 
 batch_normalization_3 (Batc  (None, 512)              2048      
 hNormalization)                                                 
                                                                 
 dropout_3 (Dropout)         (None, 512)               0         
                                                                 
 dense_1 (Dense)             (None, 12)                6156      
                                                                 
=================================================================
Total params: 9,733,804
Trainable params: 9,732,332
Non-trainable params: 1,472
_________________________________________________________________

Model7 Architecture Visualization

#visualize model7 architecture
plot_model(model7, to_file='model7_plot.png', show_shapes=True, show_layer_names=True)

#visualize model7 architecture using visualkeras
visualkeras.layered_view(model7, legend=True)

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history7=model7.fit(X_train,
          y_train_onehot,
          epochs=100,
          batch_size=32,validation_split=0.10,callbacks=[es, mc])

Epoch 1/100
120/121 [============================>.] - ETA: 0s - loss: 1.9258 - accuracy: 0.4435
Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5
121/121 [==============================] - 14s 73ms/step - loss: 1.9249 - accuracy: 0.4440 - val_loss: 23.2398 - val_accuracy: 0.1215
Epoch 2/100
120/121 [============================>.] - ETA: 0s - loss: 1.0223 - accuracy: 0.6716
Epoch 2: val_accuracy did not improve from 0.12150
121/121 [==============================] - 8s 63ms/step - loss: 1.0225 - accuracy: 0.6712 - val_loss: 18.6191 - val_accuracy: 0.1215
Epoch 3/100
120/121 [============================>.] - ETA: 0s - loss: 0.6844 - accuracy: 0.7745
Epoch 3: val_accuracy did not improve from 0.12150
121/121 [==============================] - 8s 63ms/step - loss: 0.6857 - accuracy: 0.7744 - val_loss: 21.4383 - val_accuracy: 0.1215
Epoch 4/100
120/121 [============================>.] - ETA: 0s - loss: 0.5225 - accuracy: 0.8263
Epoch 4: val_accuracy improved from 0.12150 to 0.14252, saving model to best_model.h5
121/121 [==============================] - 8s 70ms/step - loss: 0.5232 - accuracy: 0.8261 - val_loss: 7.6242 - val_accuracy: 0.1425
Epoch 5/100
120/121 [============================>.] - ETA: 0s - loss: 0.4009 - accuracy: 0.8602
Epoch 5: val_accuracy improved from 0.14252 to 0.18925, saving model to best_model.h5
121/121 [==============================] - 8s 68ms/step - loss: 0.4030 - accuracy: 0.8594 - val_loss: 5.4088 - val_accuracy: 0.1893
Epoch 6/100
120/121 [============================>.] - ETA: 0s - loss: 0.3923 - accuracy: 0.8622
Epoch 6: val_accuracy improved from 0.18925 to 0.67290, saving model to best_model.h5
121/121 [==============================] - 14s 113ms/step - loss: 0.3940 - accuracy: 0.8615 - val_loss: 1.0581 - val_accuracy: 0.6729
Epoch 7/100
120/121 [============================>.] - ETA: 0s - loss: 0.3071 - accuracy: 0.8935
Epoch 7: val_accuracy improved from 0.67290 to 0.79673, saving model to best_model.h5
121/121 [==============================] - 8s 69ms/step - loss: 0.3068 - accuracy: 0.8937 - val_loss: 0.7298 - val_accuracy: 0.7967
Epoch 8/100
120/121 [============================>.] - ETA: 0s - loss: 0.2381 - accuracy: 0.9133
Epoch 8: val_accuracy did not improve from 0.79673
121/121 [==============================] - 8s 62ms/step - loss: 0.2386 - accuracy: 0.9132 - val_loss: 1.3537 - val_accuracy: 0.6612
Epoch 9/100
120/121 [============================>.] - ETA: 0s - loss: 0.1960 - accuracy: 0.9320
Epoch 9: val_accuracy improved from 0.79673 to 0.79907, saving model to best_model.h5
121/121 [==============================] - 8s 68ms/step - loss: 0.1963 - accuracy: 0.9319 - val_loss: 0.7284 - val_accuracy: 0.7991
Epoch 10/100
120/121 [============================>.] - ETA: 0s - loss: 0.1645 - accuracy: 0.9409
Epoch 10: val_accuracy did not improve from 0.79907
121/121 [==============================] - 8s 62ms/step - loss: 0.1655 - accuracy: 0.9402 - val_loss: 0.8174 - val_accuracy: 0.7617
Epoch 11/100
120/121 [============================>.] - ETA: 0s - loss: 0.1333 - accuracy: 0.9555
Epoch 11: val_accuracy improved from 0.79907 to 0.85047, saving model to best_model.h5
121/121 [==============================] - 8s 67ms/step - loss: 0.1351 - accuracy: 0.9550 - val_loss: 0.4578 - val_accuracy: 0.8505
Epoch 12/100
120/121 [============================>.] - ETA: 0s - loss: 0.1134 - accuracy: 0.9591
Epoch 12: val_accuracy did not improve from 0.85047
121/121 [==============================] - 8s 63ms/step - loss: 0.1148 - accuracy: 0.9584 - val_loss: 1.0042 - val_accuracy: 0.7593
Epoch 13/100
120/121 [============================>.] - ETA: 0s - loss: 0.1403 - accuracy: 0.9497
Epoch 13: val_accuracy did not improve from 0.85047
121/121 [==============================] - 8s 65ms/step - loss: 0.1414 - accuracy: 0.9493 - val_loss: 0.6063 - val_accuracy: 0.8458
Epoch 14/100
120/121 [============================>.] - ETA: 0s - loss: 0.1181 - accuracy: 0.9583
Epoch 14: val_accuracy did not improve from 0.85047
121/121 [==============================] - 8s 64ms/step - loss: 0.1183 - accuracy: 0.9581 - val_loss: 1.3537 - val_accuracy: 0.6682
Epoch 15/100
120/121 [============================>.] - ETA: 0s - loss: 0.1149 - accuracy: 0.9628
Epoch 15: val_accuracy did not improve from 0.85047
121/121 [==============================] - 8s 63ms/step - loss: 0.1179 - accuracy: 0.9623 - val_loss: 1.4002 - val_accuracy: 0.6799
Epoch 16/100
120/121 [============================>.] - ETA: 0s - loss: 0.0929 - accuracy: 0.9680
Epoch 16: val_accuracy did not improve from 0.85047
121/121 [==============================] - 8s 64ms/step - loss: 0.0929 - accuracy: 0.9680 - val_loss: 0.7927 - val_accuracy: 0.8201
Epoch 17/100
120/121 [============================>.] - ETA: 0s - loss: 0.0706 - accuracy: 0.9747
Epoch 17: val_accuracy improved from 0.85047 to 0.85280, saving model to best_model.h5
121/121 [==============================] - 9s 72ms/step - loss: 0.0705 - accuracy: 0.9748 - val_loss: 0.5961 - val_accuracy: 0.8528
Epoch 18/100
120/121 [============================>.] - ETA: 0s - loss: 0.0556 - accuracy: 0.9815
Epoch 18: val_accuracy did not improve from 0.85280
121/121 [==============================] - 8s 64ms/step - loss: 0.0561 - accuracy: 0.9813 - val_loss: 1.0609 - val_accuracy: 0.7313
Epoch 19/100
120/121 [============================>.] - ETA: 0s - loss: 0.1063 - accuracy: 0.9648
Epoch 19: val_accuracy did not improve from 0.85280
121/121 [==============================] - 8s 64ms/step - loss: 0.1063 - accuracy: 0.9649 - val_loss: 0.5733 - val_accuracy: 0.8458
Epoch 20/100
120/121 [============================>.] - ETA: 0s - loss: 0.0721 - accuracy: 0.9755
Epoch 20: val_accuracy did not improve from 0.85280
121/121 [==============================] - 8s 63ms/step - loss: 0.0728 - accuracy: 0.9753 - val_loss: 0.5768 - val_accuracy: 0.8528
Epoch 21/100
120/121 [============================>.] - ETA: 0s - loss: 0.0902 - accuracy: 0.9714
Epoch 21: val_accuracy did not improve from 0.85280
121/121 [==============================] - 8s 64ms/step - loss: 0.0900 - accuracy: 0.9714 - val_loss: 0.6353 - val_accuracy: 0.8248
Epoch 21: early stopping

Training stopped at epoch 27. The highest validation accuracy was 0.8578 with validation loss of 0.25

Model7 Accuracy vs Epoch curve

plt.plot(history7.history['accuracy'])
plt.plot(history7.history['val_accuracy'])
plt.title('model7 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

Model is overfitting

Model7 Evaluation of model model.evaluate using the Test data

# Evaluate the model on the test data
score7 = model7.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score7[0])
print('Test accuracy:', score7[1])

Test loss: 0.7704196572303772
Test accuracy: 0.7957894802093506

# Test Prediction
y_pred = model7.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)

15/15 [==============================] - 0s 13ms/step

# Test Accuracy
accuracyScore7 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore7)

0.7957894736842105

Classification Report

# Compute the classification report
cr7 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

print('Classification report:\n',cr7)

Classification report:
                            precision    recall  f1-score   support

Small-flowered Cranesbill       0.89      0.80      0.84        50
                  Fat Hen       0.75      0.96      0.84        48
          Shepherds Purse       0.78      0.61      0.68        23
             Common wheat       0.76      0.59      0.67        22
         Common Chickweed       0.94      0.80      0.87        61
                 Charlock       0.84      0.95      0.89        39
                 Cleavers       0.87      0.90      0.88        29
        Scentless Mayweed       0.85      0.87      0.86        52
               Sugar beet       0.82      0.84      0.83        38
                    Maize       0.83      0.68      0.75        22
              Black-grass       0.38      0.46      0.41        26
         Loose Silky-bent       0.74      0.75      0.75        65

                 accuracy                           0.80       475
                macro avg       0.79      0.77      0.77       475
             weighted avg       0.81      0.80      0.80       475

The accuracy of the model is very low at 0.24. For most of the classes, precision, recall, and f1-score are also very low. This suggests that the model is not able to classify the images correctly, and the predictions made by the model are almost random. Therefore, it is not a valid model.

Scoresdf = Scoresdf.append({'model':'Model7','accuracy':accuracyScore7,'layers':'20','total parameters':'9,733,804', 'neurons':'512', 'optimizer':'Adam'},ignore_index=True)

Confusion Matrix

# Compute the Confusion matrix
cm7 = confusion_matrix(normal_y_test, y_test_pred_classes)

plt.figure(figsize=(8,6))
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
sns.heatmap(cm7, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.show()

Model7 is not a valid model.

Model8

Continuing trying to improve the previous model, we changed the number of layers and neurons.

• Convolutional layer, 64 filters, kernel (3,3), ReLU
• Batch normalization
• MaxPooling2D
• Convolutional layer, 128 filters, kernel (3,3).
• Batch normalization
• MaxPooling2D
• Convolutional layer, 256 filters, kernel (3,3).
• Batch normalization
• MaxPooling2D
• Convolutional layer, 128 filters, kernel (3,3).
• Batch normalization
• MaxPooling2D
• Convolutional layer, 64 filters, kernel (5,5).
• Batch normalization
• MaxPooling2D
• Flatten Layer
• Dense Layer, 256 neurons
• BatchNormalization
• Drop Layer 0.5

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

Model Implementation

# initialized a sequential model
model8 = Sequential()

model8.add(Conv2D(64, kernel_size=(3,3), activation='relu', input_shape=input_shape))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))

model8.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))

model8.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))

model8.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))

model8.add(Conv2D(64, kernel_size=(5,5), activation='relu'))
model8.add(BatchNormalization())
model8.add(MaxPooling2D(pool_size=(2,2)))

model8.add(Flatten())
model8.add(Dense(256, activation='relu'))
model8.add(BatchNormalization())
model8.add(Dropout(0.5))
model8.add(Dense(num_classes, activation='softmax'))

# Compile the model
model8.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# Print the model summary
model8.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 126, 126, 64)      1792      
                                                                 
 batch_normalization (BatchN  (None, 126, 126, 64)     256       
 ormalization)                                                   
                                                                 
 max_pooling2d (MaxPooling2D  (None, 63, 63, 64)       0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 61, 61, 128)       73856     
                                                                 
 batch_normalization_1 (Batc  (None, 61, 61, 128)      512       
 hNormalization)                                                 
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 30, 30, 128)      0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 28, 28, 256)       295168    
                                                                 
 batch_normalization_2 (Batc  (None, 28, 28, 256)      1024      
 hNormalization)                                                 
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 14, 14, 256)      0         
 2D)                                                             
                                                                 
 conv2d_3 (Conv2D)           (None, 12, 12, 128)       295040    
                                                                 
 batch_normalization_3 (Batc  (None, 12, 12, 128)      512       
 hNormalization)                                                 
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 6, 6, 128)        0         
 2D)                                                             
                                                                 
 conv2d_4 (Conv2D)           (None, 2, 2, 64)          204864    
                                                                 
 batch_normalization_4 (Batc  (None, 2, 2, 64)         256       
 hNormalization)                                                 
                                                                 
 max_pooling2d_4 (MaxPooling  (None, 1, 1, 64)         0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 64)                0         
                                                                 
 dense (Dense)               (None, 256)               16640     
                                                                 
 batch_normalization_5 (Batc  (None, 256)              1024      
 hNormalization)                                                 
                                                                 
 dropout (Dropout)           (None, 256)               0         
                                                                 
 dense_1 (Dense)             (None, 12)                3084      
                                                                 
=================================================================
Total params: 894,028
Trainable params: 892,236
Non-trainable params: 1,792
_________________________________________________________________

Model8 Architecture Visualization

#visualize model4 architecture
plot_model(model8, to_file='model8_plot.png', show_shapes=True, show_layer_names=True)

#visualize model8 architecture using visualkeras
visualkeras.layered_view(model8, legend=True)

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history8=model8.fit(X_train,
          y_train_onehot,
          epochs=100,
          batch_size=32,validation_split=0.10,callbacks=[es, mc])

Epoch 1/100
121/121 [==============================] - ETA: 0s - loss: 1.9141 - accuracy: 0.4328
Epoch 1: val_accuracy improved from -inf to 0.11916, saving model to best_model.h5
121/121 [==============================] - 12s 51ms/step - loss: 1.9141 - accuracy: 0.4328 - val_loss: 6.6545 - val_accuracy: 0.1192
Epoch 2/100
121/121 [==============================] - ETA: 0s - loss: 1.0103 - accuracy: 0.6740
Epoch 2: val_accuracy improved from 0.11916 to 0.12150, saving model to best_model.h5
121/121 [==============================] - 6s 49ms/step - loss: 1.0103 - accuracy: 0.6740 - val_loss: 13.4881 - val_accuracy: 0.1215
Epoch 3/100
120/121 [============================>.] - ETA: 0s - loss: 0.7239 - accuracy: 0.7560
Epoch 3: val_accuracy improved from 0.12150 to 0.12850, saving model to best_model.h5
121/121 [==============================] - 6s 49ms/step - loss: 0.7257 - accuracy: 0.7554 - val_loss: 9.6117 - val_accuracy: 0.1285
Epoch 4/100
120/121 [============================>.] - ETA: 0s - loss: 0.5796 - accuracy: 0.8023
Epoch 4: val_accuracy did not improve from 0.12850
121/121 [==============================] - 6s 48ms/step - loss: 0.5814 - accuracy: 0.8019 - val_loss: 8.5606 - val_accuracy: 0.1285
Epoch 5/100
120/121 [============================>.] - ETA: 0s - loss: 0.4688 - accuracy: 0.8372
Epoch 5: val_accuracy improved from 0.12850 to 0.43692, saving model to best_model.h5
121/121 [==============================] - 6s 49ms/step - loss: 0.4694 - accuracy: 0.8368 - val_loss: 2.2889 - val_accuracy: 0.4369
Epoch 6/100
121/121 [==============================] - ETA: 0s - loss: 0.4436 - accuracy: 0.8417
Epoch 6: val_accuracy improved from 0.43692 to 0.51636, saving model to best_model.h5
121/121 [==============================] - 6s 49ms/step - loss: 0.4436 - accuracy: 0.8417 - val_loss: 1.5959 - val_accuracy: 0.5164
Epoch 7/100
120/121 [============================>.] - ETA: 0s - loss: 0.3202 - accuracy: 0.8878
Epoch 7: val_accuracy improved from 0.51636 to 0.69860, saving model to best_model.h5
121/121 [==============================] - 6s 50ms/step - loss: 0.3200 - accuracy: 0.8880 - val_loss: 1.0497 - val_accuracy: 0.6986
Epoch 8/100
121/121 [==============================] - ETA: 0s - loss: 0.3005 - accuracy: 0.8942
Epoch 8: val_accuracy did not improve from 0.69860
121/121 [==============================] - 6s 48ms/step - loss: 0.3005 - accuracy: 0.8942 - val_loss: 5.7750 - val_accuracy: 0.2056
Epoch 9/100
120/121 [============================>.] - ETA: 0s - loss: 0.2668 - accuracy: 0.9023
Epoch 9: val_accuracy did not improve from 0.69860
121/121 [==============================] - 6s 49ms/step - loss: 0.2665 - accuracy: 0.9025 - val_loss: 7.3724 - val_accuracy: 0.2126
Epoch 10/100
121/121 [==============================] - ETA: 0s - loss: 0.2086 - accuracy: 0.9194
Epoch 10: val_accuracy improved from 0.69860 to 0.73598, saving model to best_model.h5
121/121 [==============================] - 6s 50ms/step - loss: 0.2086 - accuracy: 0.9194 - val_loss: 0.8748 - val_accuracy: 0.7360
Epoch 11/100
120/121 [============================>.] - ETA: 0s - loss: 0.2114 - accuracy: 0.9227
Epoch 11: val_accuracy improved from 0.73598 to 0.82944, saving model to best_model.h5
121/121 [==============================] - 6s 48ms/step - loss: 0.2125 - accuracy: 0.9223 - val_loss: 0.5081 - val_accuracy: 0.8294
Epoch 12/100
120/121 [============================>.] - ETA: 0s - loss: 0.2021 - accuracy: 0.9268
Epoch 12: val_accuracy improved from 0.82944 to 0.84813, saving model to best_model.h5
121/121 [==============================] - 6s 52ms/step - loss: 0.2023 - accuracy: 0.9267 - val_loss: 0.4966 - val_accuracy: 0.8481
Epoch 13/100
121/121 [==============================] - ETA: 0s - loss: 0.1775 - accuracy: 0.9363
Epoch 13: val_accuracy did not improve from 0.84813
121/121 [==============================] - 6s 49ms/step - loss: 0.1775 - accuracy: 0.9363 - val_loss: 1.5754 - val_accuracy: 0.6519
Epoch 14/100
120/121 [============================>.] - ETA: 0s - loss: 0.1478 - accuracy: 0.9443
Epoch 14: val_accuracy did not improve from 0.84813
121/121 [==============================] - 6s 49ms/step - loss: 0.1481 - accuracy: 0.9441 - val_loss: 0.5397 - val_accuracy: 0.8435
Epoch 15/100
121/121 [==============================] - ETA: 0s - loss: 0.1143 - accuracy: 0.9615
Epoch 15: val_accuracy did not improve from 0.84813
121/121 [==============================] - 6s 50ms/step - loss: 0.1143 - accuracy: 0.9615 - val_loss: 1.3775 - val_accuracy: 0.6916
Epoch 16/100
120/121 [============================>.] - ETA: 0s - loss: 0.2131 - accuracy: 0.9289
Epoch 16: val_accuracy did not improve from 0.84813
121/121 [==============================] - 6s 47ms/step - loss: 0.2155 - accuracy: 0.9288 - val_loss: 1.4203 - val_accuracy: 0.6799
Epoch 17/100
121/121 [==============================] - ETA: 0s - loss: 0.1556 - accuracy: 0.9480
Epoch 17: val_accuracy improved from 0.84813 to 0.86682, saving model to best_model.h5
121/121 [==============================] - 6s 52ms/step - loss: 0.1556 - accuracy: 0.9480 - val_loss: 0.4611 - val_accuracy: 0.8668
Epoch 18/100
120/121 [============================>.] - ETA: 0s - loss: 0.0900 - accuracy: 0.9701
Epoch 18: val_accuracy did not improve from 0.86682
121/121 [==============================] - 6s 48ms/step - loss: 0.0899 - accuracy: 0.9701 - val_loss: 0.8230 - val_accuracy: 0.7477
Epoch 19/100
120/121 [============================>.] - ETA: 0s - loss: 0.0837 - accuracy: 0.9714
Epoch 19: val_accuracy did not improve from 0.86682
121/121 [==============================] - 6s 49ms/step - loss: 0.0856 - accuracy: 0.9711 - val_loss: 1.8179 - val_accuracy: 0.6379
Epoch 20/100
121/121 [==============================] - ETA: 0s - loss: 0.1230 - accuracy: 0.9561
Epoch 20: val_accuracy did not improve from 0.86682
121/121 [==============================] - 6s 50ms/step - loss: 0.1230 - accuracy: 0.9561 - val_loss: 0.7045 - val_accuracy: 0.8061
Epoch 21/100
120/121 [============================>.] - ETA: 0s - loss: 0.1514 - accuracy: 0.9479
Epoch 21: val_accuracy did not improve from 0.86682
121/121 [==============================] - 6s 48ms/step - loss: 0.1512 - accuracy: 0.9480 - val_loss: 0.6942 - val_accuracy: 0.7944
Epoch 22/100
120/121 [============================>.] - ETA: 0s - loss: 0.0827 - accuracy: 0.9693
Epoch 22: val_accuracy did not improve from 0.86682
121/121 [==============================] - 6s 50ms/step - loss: 0.0827 - accuracy: 0.9693 - val_loss: 0.6314 - val_accuracy: 0.8341
Epoch 23/100
121/121 [==============================] - ETA: 0s - loss: 0.0603 - accuracy: 0.9800
Epoch 23: val_accuracy did not improve from 0.86682
121/121 [==============================] - 6s 49ms/step - loss: 0.0603 - accuracy: 0.9800 - val_loss: 2.7126 - val_accuracy: 0.5047
Epoch 24/100
120/121 [============================>.] - ETA: 0s - loss: 0.0530 - accuracy: 0.9846
Epoch 24: val_accuracy improved from 0.86682 to 0.89720, saving model to best_model.h5
121/121 [==============================] - 6s 49ms/step - loss: 0.0538 - accuracy: 0.9844 - val_loss: 0.3790 - val_accuracy: 0.8972
Epoch 25/100
120/121 [============================>.] - ETA: 0s - loss: 0.1026 - accuracy: 0.9643
Epoch 25: val_accuracy did not improve from 0.89720
121/121 [==============================] - 6s 49ms/step - loss: 0.1034 - accuracy: 0.9641 - val_loss: 1.1961 - val_accuracy: 0.6776
Epoch 26/100
121/121 [==============================] - ETA: 0s - loss: 0.1000 - accuracy: 0.9670
Epoch 26: val_accuracy did not improve from 0.89720
121/121 [==============================] - 6s 48ms/step - loss: 0.1000 - accuracy: 0.9670 - val_loss: 0.5659 - val_accuracy: 0.8621
Epoch 27/100
120/121 [============================>.] - ETA: 0s - loss: 0.0665 - accuracy: 0.9784
Epoch 27: val_accuracy did not improve from 0.89720
121/121 [==============================] - 6s 49ms/step - loss: 0.0669 - accuracy: 0.9782 - val_loss: 0.5263 - val_accuracy: 0.8668
Epoch 28/100
121/121 [==============================] - ETA: 0s - loss: 0.0534 - accuracy: 0.9839
Epoch 28: val_accuracy did not improve from 0.89720
121/121 [==============================] - 6s 48ms/step - loss: 0.0534 - accuracy: 0.9839 - val_loss: 0.5661 - val_accuracy: 0.8668
Epoch 29/100
120/121 [============================>.] - ETA: 0s - loss: 0.0806 - accuracy: 0.9729
Epoch 29: val_accuracy did not improve from 0.89720
121/121 [==============================] - 6s 47ms/step - loss: 0.0826 - accuracy: 0.9724 - val_loss: 1.1458 - val_accuracy: 0.7033
Epoch 30/100
120/121 [============================>.] - ETA: 0s - loss: 0.0935 - accuracy: 0.9682
Epoch 30: val_accuracy did not improve from 0.89720
121/121 [==============================] - 6s 49ms/step - loss: 0.0937 - accuracy: 0.9680 - val_loss: 0.6199 - val_accuracy: 0.8528
Epoch 31/100
120/121 [============================>.] - ETA: 0s - loss: 0.0455 - accuracy: 0.9870
Epoch 31: val_accuracy did not improve from 0.89720
121/121 [==============================] - 6s 47ms/step - loss: 0.0459 - accuracy: 0.9867 - val_loss: 0.6134 - val_accuracy: 0.8458
Epoch 32/100
121/121 [==============================] - ETA: 0s - loss: 0.0428 - accuracy: 0.9857
Epoch 32: val_accuracy did not improve from 0.89720
121/121 [==============================] - 6s 48ms/step - loss: 0.0428 - accuracy: 0.9857 - val_loss: 0.7207 - val_accuracy: 0.8575
Epoch 33/100
121/121 [==============================] - ETA: 0s - loss: 0.0531 - accuracy: 0.9802
Epoch 33: val_accuracy did not improve from 0.89720
121/121 [==============================] - 6s 49ms/step - loss: 0.0531 - accuracy: 0.9802 - val_loss: 0.7820 - val_accuracy: 0.7757
Epoch 34/100
120/121 [============================>.] - ETA: 0s - loss: 0.0499 - accuracy: 0.9828
Epoch 34: val_accuracy did not improve from 0.89720
121/121 [==============================] - 6s 48ms/step - loss: 0.0509 - accuracy: 0.9826 - val_loss: 1.0555 - val_accuracy: 0.7687
Epoch 34: early stopping

Training stopped at epoch 20. The highest validation accuracy was 0.8916 with a validation loss of 0.3837.

Model8 Accuracy vs Epoch curve

plt.plot(history8.history['accuracy'])
plt.plot(history8.history['val_accuracy'])
plt.title('model8 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

The model overfits.

Model8 Evaluation of model model.evaluate using the Test data

# Evaluate the model on the test data
score8 = model8.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score8[0])
print('Test accuracy:', score8[1])

Test loss: 0.9670275449752808
Test accuracy: 0.7873684167861938

# Test Prediction
y_pred = model8.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)

15/15 [==============================] - 0s 13ms/step

# Test Accuracy
accuracyScore8 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore8)

0.7873684210526316

Classification Report

# Compute the classification report
cr8 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

print('Classification report:\n',cr8)

Classification report:
                            precision    recall  f1-score   support

Small-flowered Cranesbill       0.90      0.92      0.91        50
                  Fat Hen       0.94      0.96      0.95        48
          Shepherds Purse       0.89      0.35      0.50        23
             Common wheat       0.80      0.73      0.76        22
         Common Chickweed       0.63      0.98      0.77        61
                 Charlock       0.95      0.90      0.92        39
                 Cleavers       0.81      0.90      0.85        29
        Scentless Mayweed       0.97      0.63      0.77        52
               Sugar beet       0.80      0.84      0.82        38
                    Maize       1.00      0.59      0.74        22
              Black-grass       0.39      0.62      0.48        26
         Loose Silky-bent       0.80      0.66      0.72        65

                 accuracy                           0.79       475
                macro avg       0.82      0.76      0.77       475
             weighted avg       0.83      0.79      0.79       475

Overall the accuracy is low at 0.43. We have developed better models earlier.

Scoresdf = Scoresdf.append({'model':'Model8','accuracy':accuracyScore8,'layers':'20','total parameters':'894,028', 'neurons':'256', 'optimizer':'Adam'},ignore_index=True)

Confusion Matrix

# Compute Confusion matrix
cm8 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm8, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()

From the confusion matrix we can observe that Model8 is unable to predict many classes correctly at all. Overall, this is not a good model.

Model 9

backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

This will be our final model trying to improve our previous models. Again, we will modify the number of layers, filter size and neurons.

Model Implementation

# Intializing a sequential model
model9 = Sequential()

# Adding first conv layer with 64 filters and kernel size 3x3 , padding 'same' provides the output size same as the input size
# Input_shape denotes input image dimension of MNIST images
model9.add(Conv2D(64, (3, 3), activation='relu', padding="same", input_shape=input_shape))
model9.add(Conv2D(64, (3, 3), activation='relu', padding="same"))
# Adding max pooling to reduce the size of output of first conv layer
model9.add(MaxPooling2D((2, 2), padding = 'same'))

model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
model9.add(MaxPooling2D((2, 2), padding = 'same'))
model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
model9.add(MaxPooling2D((2, 2), padding = 'same'))

# flattening the output of the conv layer after max pooling to make it ready for creating dense connections
model9.add(Flatten())

# Adding a fully connected dense layer with 100 neurons
model9.add(Dense(128, activation='relu'))
model9.add(Dropout(0.25))
model9.add(Dense(64, activation='relu'))
# Adding the output layer with 10 neurons and activation functions as softmax since this is a multi-class classification problem
model9.add(Dense(num_classes, activation='softmax'))

# Using Adam Optimizer
opt = Adam()

# Compile model
model9.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

model9.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 128, 128, 64)      1792      
                                                                 
 conv2d_1 (Conv2D)           (None, 128, 128, 64)      36928     
                                                                 
 max_pooling2d (MaxPooling2D  (None, 64, 64, 64)       0         
 )                                                               
                                                                 
 conv2d_2 (Conv2D)           (None, 64, 64, 32)        18464     
                                                                 
 conv2d_3 (Conv2D)           (None, 64, 64, 32)        9248      
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 32, 32, 32)       0         
 2D)                                                             
                                                                 
 conv2d_4 (Conv2D)           (None, 32, 32, 32)        9248      
                                                                 
 conv2d_5 (Conv2D)           (None, 32, 32, 32)        9248      
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 16, 16, 32)       0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 8192)              0         
                                                                 
 dense (Dense)               (None, 128)               1048704   
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_1 (Dense)             (None, 64)                8256      
                                                                 
 dense_2 (Dense)             (None, 12)                780       
                                                                 
=================================================================
Total params: 1,142,668
Trainable params: 1,142,668
Non-trainable params: 0
_________________________________________________________________

Model9 Architecture Visualization

#visualize model9 architecture
plot_model(model9, to_file='model9_plot.png', show_shapes=True, show_layer_names=True)

#visualize model9 architecture using visualkeras
visualkeras.layered_view(model9, legend=True)

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
# Fitting the model with 100 epochs and validation_split as 10%
history9=model9.fit(X_train,
          y_train_onehot,
          epochs=100,
          batch_size=32,validation_split=0.10,callbacks=[es, mc])

Epoch 1/100
120/121 [============================>.] - ETA: 0s - loss: 2.4315 - accuracy: 0.1240
Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5
121/121 [==============================] - 10s 55ms/step - loss: 2.4312 - accuracy: 0.1237 - val_loss: 2.4521 - val_accuracy: 0.1215
Epoch 2/100
120/121 [============================>.] - ETA: 0s - loss: 2.0708 - accuracy: 0.2891
Epoch 2: val_accuracy improved from 0.12150 to 0.35748, saving model to best_model.h5
121/121 [==============================] - 6s 50ms/step - loss: 2.0710 - accuracy: 0.2893 - val_loss: 1.7058 - val_accuracy: 0.3575
Epoch 3/100
120/121 [============================>.] - ETA: 0s - loss: 1.5718 - accuracy: 0.4453
Epoch 3: val_accuracy improved from 0.35748 to 0.48131, saving model to best_model.h5
121/121 [==============================] - 6s 52ms/step - loss: 1.5718 - accuracy: 0.4453 - val_loss: 1.5090 - val_accuracy: 0.4813
Epoch 4/100
120/121 [============================>.] - ETA: 0s - loss: 1.3208 - accuracy: 0.5378
Epoch 4: val_accuracy improved from 0.48131 to 0.59112, saving model to best_model.h5
121/121 [==============================] - 6s 50ms/step - loss: 1.3205 - accuracy: 0.5378 - val_loss: 1.1847 - val_accuracy: 0.5911
Epoch 5/100
121/121 [==============================] - ETA: 0s - loss: 1.0526 - accuracy: 0.6356
Epoch 5: val_accuracy improved from 0.59112 to 0.62617, saving model to best_model.h5
121/121 [==============================] - 6s 50ms/step - loss: 1.0526 - accuracy: 0.6356 - val_loss: 1.0737 - val_accuracy: 0.6262
Epoch 6/100
121/121 [==============================] - ETA: 0s - loss: 0.8805 - accuracy: 0.6868
Epoch 6: val_accuracy improved from 0.62617 to 0.67290, saving model to best_model.h5
121/121 [==============================] - 6s 51ms/step - loss: 0.8805 - accuracy: 0.6868 - val_loss: 0.9630 - val_accuracy: 0.6729
Epoch 7/100
120/121 [============================>.] - ETA: 0s - loss: 0.7282 - accuracy: 0.7346
Epoch 7: val_accuracy improved from 0.67290 to 0.68224, saving model to best_model.h5
121/121 [==============================] - 6s 50ms/step - loss: 0.7284 - accuracy: 0.7349 - val_loss: 0.9714 - val_accuracy: 0.6822
Epoch 8/100
120/121 [============================>.] - ETA: 0s - loss: 0.5953 - accuracy: 0.7810
Epoch 8: val_accuracy improved from 0.68224 to 0.71262, saving model to best_model.h5
121/121 [==============================] - 6s 53ms/step - loss: 0.5949 - accuracy: 0.7809 - val_loss: 0.8997 - val_accuracy: 0.7126
Epoch 9/100
121/121 [==============================] - ETA: 0s - loss: 0.4768 - accuracy: 0.8095
Epoch 9: val_accuracy improved from 0.71262 to 0.71495, saving model to best_model.h5
121/121 [==============================] - 6s 50ms/step - loss: 0.4768 - accuracy: 0.8095 - val_loss: 1.0595 - val_accuracy: 0.7150
Epoch 10/100
121/121 [==============================] - ETA: 0s - loss: 0.4320 - accuracy: 0.8303
Epoch 10: val_accuracy improved from 0.71495 to 0.73364, saving model to best_model.h5
121/121 [==============================] - 6s 50ms/step - loss: 0.4320 - accuracy: 0.8303 - val_loss: 1.0687 - val_accuracy: 0.7336
Epoch 11/100
120/121 [============================>.] - ETA: 0s - loss: 0.4239 - accuracy: 0.8435
Epoch 11: val_accuracy did not improve from 0.73364
121/121 [==============================] - 6s 50ms/step - loss: 0.4241 - accuracy: 0.8433 - val_loss: 1.0077 - val_accuracy: 0.7313
Epoch 12/100
120/121 [============================>.] - ETA: 0s - loss: 0.3202 - accuracy: 0.8776
Epoch 12: val_accuracy did not improve from 0.73364
121/121 [==============================] - 6s 48ms/step - loss: 0.3203 - accuracy: 0.8773 - val_loss: 1.1118 - val_accuracy: 0.7220
Epoch 13/100
120/121 [============================>.] - ETA: 0s - loss: 0.3107 - accuracy: 0.8802
Epoch 13: val_accuracy improved from 0.73364 to 0.75467, saving model to best_model.h5
121/121 [==============================] - 6s 53ms/step - loss: 0.3113 - accuracy: 0.8799 - val_loss: 0.9295 - val_accuracy: 0.7547
Epoch 14/100
120/121 [============================>.] - ETA: 0s - loss: 0.2699 - accuracy: 0.9031
Epoch 14: val_accuracy did not improve from 0.75467
121/121 [==============================] - 6s 49ms/step - loss: 0.2700 - accuracy: 0.9030 - val_loss: 1.0213 - val_accuracy: 0.7383
Epoch 15/100
120/121 [============================>.] - ETA: 0s - loss: 0.2276 - accuracy: 0.9146
Epoch 15: val_accuracy did not improve from 0.75467
121/121 [==============================] - 6s 51ms/step - loss: 0.2274 - accuracy: 0.9147 - val_loss: 1.1179 - val_accuracy: 0.7336
Epoch 16/100
121/121 [==============================] - ETA: 0s - loss: 0.2142 - accuracy: 0.9254
Epoch 16: val_accuracy did not improve from 0.75467
121/121 [==============================] - 6s 50ms/step - loss: 0.2142 - accuracy: 0.9254 - val_loss: 1.1452 - val_accuracy: 0.7173
Epoch 17/100
120/121 [============================>.] - ETA: 0s - loss: 0.1628 - accuracy: 0.9440
Epoch 17: val_accuracy did not improve from 0.75467
121/121 [==============================] - 6s 50ms/step - loss: 0.1625 - accuracy: 0.9441 - val_loss: 1.0877 - val_accuracy: 0.7360
Epoch 18/100
120/121 [============================>.] - ETA: 0s - loss: 0.1426 - accuracy: 0.9451
Epoch 18: val_accuracy did not improve from 0.75467
121/121 [==============================] - 6s 52ms/step - loss: 0.1425 - accuracy: 0.9452 - val_loss: 1.2663 - val_accuracy: 0.7430
Epoch 18: early stopping

Training stopped at epoch 20. The highest validation accuracy was 0.79907 with a validation loss of 1.145.

Model9 Accuracy vs Epoch curve

plt.plot(history9.history['accuracy'])
plt.plot(history9.history['val_accuracy'])
plt.title('model9 accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

The model overfits.

Model9 Evaluation of model model.evaluate using the Test data

# Evaluate the model on the test data
score9 = model9.evaluate(X_test, y_test_onehot, verbose=0)
print('Test loss:', score8[0])
print('Test accuracy:', score8[1])

Test loss: 0.9670275449752808
Test accuracy: 0.7873684167861938

# Test Prediction
y_pred = model9.predict(X_test)
y_test_pred_classes = np.argmax(y_pred, axis=1)
normal_y_test = np.argmax(y_test_onehot, axis=1)

15/15 [==============================] - 0s 14ms/step

# Test Accuracy
accuracyScore9 = accuracy_score((normal_y_test), y_test_pred_classes)
print(accuracyScore8)

0.7873684210526316

Classification Report

# Compute the classification report
cr9 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

print('Classification report:\n',cr9)

Classification report:
                            precision    recall  f1-score   support

Small-flowered Cranesbill       0.90      0.86      0.88        50
                  Fat Hen       0.73      0.85      0.79        48
          Shepherds Purse       0.53      0.35      0.42        23
             Common wheat       0.64      0.64      0.64        22
         Common Chickweed       0.75      0.92      0.82        61
                 Charlock       0.77      0.87      0.82        39
                 Cleavers       0.81      0.86      0.83        29
        Scentless Mayweed       0.71      0.65      0.68        52
               Sugar beet       0.77      0.63      0.70        38
                    Maize       0.65      0.50      0.56        22
              Black-grass       0.36      0.35      0.35        26
         Loose Silky-bent       0.70      0.68      0.69        65

                 accuracy                           0.72       475
                macro avg       0.69      0.68      0.68       475
             weighted avg       0.72      0.72      0.72       475

Model9 with four convolutional layers indicates that the overall performance is good, with an accuracy of 78%.

Scoresdf = Scoresdf.append({'model':'Model9','accuracy':accuracyScore9,'layers':'14','total parameters':'1,142,668', 'neurons':'64', 'optimizer':'Adam'},ignore_index=True)

Confusion Matrix

# Compute Confusion matrix
cm9 = confusion_matrix(normal_y_test, y_test_pred_classes)
plt.figure(figsize=(8,6))
sns.heatmap(cm9, xticklabels=name_classes, yticklabels=name_classes, annot=True)
plt.xlabel("False Negatives (FN)")
plt.ylabel("False Positives (FP)")
plt.show()

Model9 performs well but not as good as Model4.

Model Improvement and Model Comparison

We tried to improve our model by implementing 9 models. One approach to implement a model is by changing the hyperparameters like optimizer, number of neurons, number of filters, adding convolution layers, batchnormalizers, changing the dropout rate, etc.

We performed these to improve our model's performance:

• Architecture Modifications: Modifying the architecture of the CNN by adding or removing layers, changing the number of filters or neurons in each layer, or using different activation functions can improve performance. We implemented diverse models this way.

• Hyperparameter Tuning: Optimizing the hyperparameters of the model, such as learning rate, batch size, and optimizer, can improve performance. This can be done using techniques such as grid search or random search. We changed the optimizer (Adam, SGD).

• Regularization: Regularization techniques such as dropout and L2 regularization can prevent overfitting and improve generalization performance. We performed dropout.

• Batch Normalization: Batch normalization is a technique that normalizes the inputs of each layer to prevent internal covariate shift. This can improve performance and reduce the number of epochs needed for training. We used batch Normalization.

• Early Stopping: Early stopping is a technique that stops training the model when the validation loss stops improving. This prevents overfitting and improves generalization performance. We used Early stopping.

We didn't implement:

• Data Augmentation: Increasing the amount of data available for training the model can improve performance. Data augmentation techniques such as flipping, rotating, scaling, and cropping images can create new variations of the same images and help the model to generalize better. However, this may require computational power so it may be expensive to perform.

• Transfer Learning: Pretrained models can be used as a starting point for training a new model. Transfer learning involves using the weights of a model trained on a large dataset, such as ImageNet, to initialize the weights of a new model. This can speed up training and improve performance.

Ensembling: Ensemble methods involve combining the predictions of multiple models to improve performance. This can be done by training multiple CNN models with different hyperparameters or architectures and combining their predictions.

From Model Improvement we came up with 9 different models which are compared below:

Scoresdf

	model	accuracy	layers	total parameters	neurons	optimizer
0	Model1	0.730526	8	7,393,868	128	Adam
1	Model2	0.787368	10	3,306,188	128	Adam
2	Model3	0.772632	10	3,306,188	128	SGD
3	Model4	0.705263	17	834,252	128	Adam
4	Model5	0.755789	20	821,324	256	Adam
5	Model6	0.421053	17	26,200,716	256	Adam
6	Model7	0.795789	20	9,733,804	512	Adam
7	Model8	0.787368	20	894,028	256	Adam
8	Model9	0.722105	14	1,142,668	64	Adam

The model that performed best was Model4 with an overall accuracy of 0.83.

Model4 has 17 layers and trained 834,52 parameters. The fully connected layer had 128 neurons and the optimizer run was Adam.

The precision scores range from 0.48 to 0.95, indicating that the model's ability to correctly identify positive instances varies by class. For example, the model has high precision in identifying Charlock and Cleavers, while it has relatively lower precision in identifying Black-grass and Shepherds Purse.

The recall scores range from 0.50 to 0.97, indicating that the model's ability to correctly identify positive instances also varies by class. For example, the model has high recall in identifying Common Chickweed and Charlock, while it has relatively lower recall in identifying Black-grass and Shepherds Purse.

The F1-scores are a harmonic mean of precision and recall and range from 0.49 to 0.92. The F1-score provides a balance between precision and recall, indicating the overall performance of the model for each class.

Final Conclusions / Takeaways

The best performing model obtained has 4 Conv2D layers, 128 filters per layer, and 2 Dense layers with 256 neurons each. This model achieved an accuracy of 83% on the test set, which is a significant improvement from the baseline model.

We could further try to improve the model's performance by trying mostly:

• Increase the size of the training dataset by collecting more images or using data augmentation techniques to generate new images from the existing ones.

• Try using pre-trained models such as VGG, ResNet or Inception and fine-tune them on your dataset.

• Continue to experiment with different hyperparameters such as batch size, number of epochs, and regularization techniques to find the best combination for your model.

We should notice that building a good performing CNN model requires a lot of experimentation and fine-tuning.

Additionally keep in mind:

• Larger filter sizes may cause loss of spatial resolution, overfitting, and high computational cost.
• Having too many convolution layers can cause overfitting, vanishing gradients, cause longer training times and produce results that are difficult to interpret.
• Having too many neurons may cause overfitting and slow down the training process.
• Dropout can help prevent overfitting
• BatchNormalization can help improve performance by estabilizing the training process and allow for higher learning rates

Overall, our model performs well but it needs to be fine tuned for the classes that are most difficult to classify, in our case, Black grass.